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Highlights 

Highlights  of  GAO-07-538,  a  report  to 
congressional  committees 


BUSINESS  SYSTEMS  MODERNIZATION 

DOD  Needs  to  Fully  Define  Policies  and 
Procedures  for  Institutionally  Managing 
Investments 


Why  GAO  Did  This  Study 

In  1995,  GAO  first  designated  the 
Department  of  Defense’s  (DOD) 
business  systems  modernization 
program  as  “high-risk,”  and 
continues  to  do  so  today.  In  2004, 
Congress  passed  legislation 
reflecting  prior  GAO 
recommendations  for  DOD  to 
adopt  a  corporate  approach  to 
information  technology  (IT) 
business  system  investment 
management.  To  support  GAO’s 
legislative  mandate  to  review 
DOD’s  efforts,  GAO  assessed 
whether  the  department’s 
corporate  investment  management 
approach  comports  with  relevant 
federal  guidance.  In  doing  so,  GAO 
applied  its  IT  Investment 
Management  framework  and 
associated  methodology,  focusing 
on  the  framework’s  stages  related 
to  the  investment  management 
provisions  of  the  Clinger-Cohen  Act 
of  1996. 


What  GAO  Recommends 


GAO  recommends  that  DOD  fully 
define  the  project  and  portfolio 
management  policies  and 
procedures  discussed  in  GAO’s 
framework.  DOD  agreed  with 
GAO’s  overall  conclusions  and 
partially  agreed  with  five  of  GAO’s 
recommendations.  However,  DOD 
disagreed  with  the  remaining  four 
recommendations,  stating  that  the 
department  is,  among  other  things, 
already  meeting  the  intent  of  these 
recommendations.  GAO  does  not 
agree;  its  recommendations  focus 
on  fully  defining  policies  and 
procedures  that  satisfy  key 
practices  in  its  framework. 


www.gao.gov/cgi-bin/getrpt7GAO-07-538. 
To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 
For  more  information,  contact  Randolph  C. 
Hite  at  (202)  512-3439  or  hiter@gao.gov. 


What  GAO  Found 

DOD  has  established  the  management  structures  needed  to  effectively 
manage  its  business  system  investments,  but  it  has  not  fully  defined  many  of 
the  related  policies  and  procedures  that  GAO’s  IT  Investment  Management 
framework  defines.  Specifically,  the  department  has  defined  four  of  nine 
practices  that  call  for  project-level  policies  and  procedures,  and  one  of  the 
five  practices  that  call  for  portfolio-level  policies  and  procedures  (see 
below).  For  example,  DOD  has  established  an  enterprisewide  IT  investment 
board  responsible  for  defining  and  implementing  its  business  system 
investment  governance  process,  documented  policies  and  procedures  for 
ensuring  that  systems  support  ongoing  and  future  business  needs,  developed 
procedures  for  identifying  and  collecting  information  about  these  systems  to 
support  investment  selection  and  control,  and  assigned  responsibility  to  an 
individual  or  a  group  for  managing  the  development  and  modification  of  the 
business  system  portfolio  selection  criteria.  However,  DOD  has  not  fully 
documented  business  system  investment  policies  and  procedures  for 
directing  investment  board  operations,  selecting  new  investments, 
reselecting  ongoing  investments,  integrating  the  investment  funding  and  the 
investment  selection  processes,  and  developing  and  maintaining  a  complete 
business  system  investment  portfolio(s). 

Regarding  project-level  investment  management  practices,  DOD  officials 
said  that  these  are  performed  at  the  component  level,  and  that  departmental 
policies  and  procedures  established  for  overseeing  components’  execution 
of  these  practices  are  sufficient.  For  portfolio-level  practices,  however,  these 
officials  stated  that  they  intend  to  improve  departmental  policies  and 
procedures  for  business  system  investments  by,  for  example,  establishing  a 
single  governance  structure,  but  plans  or  time  frames  for  doing  so  have  not 
been  established.  Until  DOD  fully  defines  departmentwide  policies  and 
procedures  for  both  individual  projects  and  portfolios  of  projects,  it  risks 
selecting  and  controlling  these  business  system  investments  in  an 
inconsistent,  incomplete,  and  ad  hoc  manner,  which  in  turn  reduces  the 
chances  that  these  investments  will  meet  mission  needs  in  the  most  cost- 
effective  manner. 


Policies  and  Procedures  for  Project-Level  and  Portfolio-Level  Management 

Stage  3:  Developing  a 

Stage  2:  Building  the 
investment  foundation 

Key  practices 
executed 

complete  investment 
portfolio 

Key  practices 
executed 

Instituting  the  investment 
board 

1/2 

Defining  the  portfolio  criteria 

1/2 

Meeting  business  needs 

1/1 

Creating  the  portfolio 

0/1 

Selecting  an  investment 

0/3 

Evaluating  the  portfolio 

0/1 

Providing  investment 
oversight 

0/1 

Conducting  postimplementation 
reviews 

0/1 

Capturing  investment 
information 

2/2 

Overall 

4/9 

1/5 

Source:  GAO. 
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1 

GAO 

^Accountability  *  Integrity  *  Reliability 

United  States  Government  Accountability  Office 
Washington,  DC  20548 


May  11,  2007 

Congressional  Committees 

For  decades,  the  Department  of  Defense  (DOD)  has  been  challenged  in 
modernizing  its  timeworn  business  systems.1  In  1995,  we  designated  DOD’s 
business  systems  modernization  program  as  high  risk,  and  we  continue  to 
designate  it  as  such  today.2  As  our  research  on  public  and  private  sector 
organizations  shows,  one  essential  ingredient  to  a  successful  systems 
modernization  program  is  having  an  effective  institutional  approach  to 
managing  information  technology  (IT)  investments. 

In  May  2001,  we  recommended  that  the  department  establish  a  corporate 
approach  to  investment  control  and  decision  making.3  Between  2001  and 
2005,  we  reported  that  the  department’s  business  systems  modernization 
program  was  still  not  being  effectively  managed,4  and  we  made  additional 
investment-related  recommendations.  Congress  subsequently  included 
provisions  in  the  Ronald  W.  Reagan  National  Defense  Authorization  Act 


business  systems  are  information  systems  that  include  financial  and  nonfinancial  systems 
and  support  DOD’s  business  operations,  such  as  civilian  personnel,  finance,  health, 
logistics,  military  personnel,  procurement,  and  transportation. 

2GAO,  High-Risk  Series:  An  Update,  GAO-07-310  (Washington,  D.C.:  January  2007). 

3GAO,  Information  Technology:  Architecture  Needed  to  Guide  Modernization  of  DOD’s 
Financial  Operations,  GAO-Ol-525  (Washington,  D.C.:  May  17,  2001). 

4See,  for  example,  GAO,  DOD  Business  Systems  Modernization:  Long-standing 
Weaknesses  in  Enterprise  Architecture  Development  Need  to  Be  Addressed,  GAO-05-702 
(Washington,  D.C.:  July  22,  2005);  DOD  Business  Systems  Modernization:  Billions  Being 
Invested  without  Adequate  Oversight,  GAO-05-381  (Washington,  D.C.:  Apr.  29,  2005);  DOD 
Business  Systems  Modernization:  Limited  Progress  in  Development  of  Business 
Enterprise  Architecture  and  Oversight  of  Information  Technology  Investments, 
GAO-04-731R  (Washington,  D.C.:  May  17,  2004);  DOD  Business  Systems  Modernization: 
Important  Progress  Made  to  Develop  Business  Enterprise  Architecture,  but  Much  Work 
Remains,  GAO-03-1018  (Washington,  D.C.:  Sept.  19,  2003);  Business  Systems 
Modernization:  Summary  of  GAO’s  Assessment  of  the  Department  of  Defense’s  Initial 
Business  Enterprise  Architecture,  GAO-03-877R  (Washington,  D.C.:  July  7,  2003); 
Information  Technology:  Observations  on  Department  of  Defense’s  Draft  Enterprise 
Architecture,  GAO-03-571R  (Washington,  D.C.:  Mar.  28,  2003);  DOD  Business  Systems 
Modernization:  Improvements  to  Enterprise  Architecture  Development  and 
Implementation  Efforts  Needed,  GAO-03-458  (Washington,  D.C.:  Feb.  28,  2003);  and 
GAO-Ol-525. 
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for  Fiscal  Year  20055  that  reflected  our  recommendations,  including  those 
for  establishing  and  implementing  effective  business  system  investment 
management  structures  and  processes. 

Between  2005  and  2006, 6  we  reported  that  DOD  had  made  important 
progress  in  establishing  and  implementing  these  structures  and  processes, 
but  that  much  remained  to  be  accomplished  relative  to  the  act’s 
requirements.  For  example,  we  reported  that  the  department’s  business 
system  investment  approach  was  not  institutionalized  at  all  levels  of  the 
department. 

To  support  GAO’s  legislative  mandate  to  review  DOD’s  annual  report  on 
its  business  systems  modernization  program,  and  as  agreed  with  your 
offices,  the  objective  of  this  review  was  to  determine  whether  DOD’s 
corporate  investment  management  approach  comports  with  relevant 
federal  guidance.  To  accomplish  our  objective,  we  analyzed  documents 
and  interviewed  agency  officials  to  determine  whether  DOD  has  developed 
the  structures,  policies,  and  procedures  associated  with  executing  those 
key  practices  in  our  IT  Investment  Management  (ITIM)  framework  that 
assist  organizations  in  complying  with  the  investment  management 
provisions  of  the  Clinger-Cohen  Act  of  1996.7  This  framework  provides  a 
hierarchical  maturity  model  for  IT  investment  management  and  a  method 
for  evaluating  and  assessing  the  maturity  of  an  agency’s  investment 
management.  We  performed  our  work  at  DOD  headquarters  in  Arlington, 
Virginia,  from  August  2006  through  April  2007  in  accordance  with 
generally  accepted  government  auditing  standards.  Details  on  our 
objective,  scope,  and  methodology  are  contained  in  appendix  I. 


5Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No. 
108-375,  §  332,  118  Stat.  1811,  1851-1856  (Oct.  28,  2004)  (codified  in  part  at  10  U.S.C.  § 

2222). 

6GAO,  Defense  Business  Transformation:  A  Comprehensive  Plan ,  Integrated  Efforts,  and 
Sustained  Leadership  Are  Needed  to  Assure  Success,  GAO-07-229T  (Washington,  D.C.: 

Nov.  16,  2006);  Business  Systems  Modernization:  DOD  Continues  to  Improve 
Institutional  Approach,  but  Further  Steps  Needed,  GAO-06-658  (Washington,  D.C.:  May  15, 
2006);  and  DOD  Business  Systems  Modernization:  Important  Progress  Made  in 
Establishing  Foundational  Architecture  Products  and  Investment  Management 
Practices,  but  Much  Work  Remains,  GAO-06-219  (Washington,  D.C.:  Nov.  23,  2005). 

7GAO,  Information  Technology  Investment  Management:  A  Framework  for  Assessing  and 
Improving  Process  Maturity,  GAO-04-394G  (Washington,  D.C.:  March  2004). 
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Results  in  Brief 


DOD  has  established  the  management  structures  needed  to  effectively 
manage  its  business  system  investments,  but  it  has  not  fully  defined  many 
of  the  related  policies  and  procedures  that  our  framework  defines. 
Specifically,  DOD  has  fully  defined  four  of  nine  key  practices  that  call  for 
project-level  policies  and  procedures,  and  one  of  the  five  practices  that 
call  for  portfolio-level  policies  and  procedures.  For  example,  regarding 
project-level  investment,  the  department  has  (1)  established  an 
enterprisewide  investment  board  and  subordinate  boards  that  are 
responsible  for  business  system  investment  governance,  (2)  documented 
policies  and  procedures  for  ensuring  that  systems  support  ongoing  and 
future  business  needs,  (3)  developed  procedures  for  identifying  and 
collecting  information  about  these  systems  to  support  investment 
selection  and  control,  and  (4)  assigned  responsibility  for  ensuring  that  the 
information  collected  during  project  identification  meets  the  needs  of  the 
investment  management  process.  Regarding  portfolio-based  investment, 
DOD  has  assigned  responsibility  to  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  for  managing  business  system 
portfolio  selection  criteria. 

However,  DOD  has  not  fully  documented  business  system  investment 
policies  and  procedures  related  to  five  key  project-level  management 
practices.  For  example,  policies  and  procedures  do  not  (1)  define  how  the 
investment  selection,  acquisition,  and  funding  processes  are  coordinated; 
(2)  specify  how  the  full  range  of  cost,  schedule,  and  benefit  data  accessible 
by  the  Investment  Review  Boards  (IRB)  are  to  be  used  in  making  selection 
(i.e.,  certification)  decisions;  (3)  specify  how  reselection  decisions  at  the 
corporate  level  (i.e.,  annual  review  decisions)  consider  investments  that 
are  in  operations  and  maintenance;  (4)  describe  how  funding  decisions  are 
integrated  with  the  process  of  selecting  an  investment  at  the  corporate 
level;  and  (5)  provide  sufficient  oversight  and  visibility  into  component- 
level  investment  management  activities,  including  component  reviews  of 
systems  in  operations  and  maintenance.  Furthermore,  DOD  does  not  have 
documented  policies  and  procedures  for  (1)  defining  the  portfolio  criteria, 
(2)  creating  the  portfolio,  (3)  evaluating  the  portfolio,  and  (4)  conducting 
postimplementation  reviews  for  all  business  systems. 

Regarding  project-level  investment  management  practices,  DOD  officials 
stated  that  these  are  performed  at  the  component  level,  and  that 
departmental  policies  and  procedures  established  for  overseeing 
execution  of  these  practices  by  components  are  sufficient.  Regarding 
portfolio-level  practices,  however,  these  officials  stated  that  they  intend  to 
improve  departmental  policies  and  procedures  for  business  system 
investments  by,  for  example,  establishing  a  single  governance  structure, 
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but  plans  or  time  frames  for  doing  so  have  not  been  established.  According 
to  our  ITIM  framework,  adequately  documenting  both  the  policies  and  the 
associated  procedures  that  govern  how  an  organization  manages  its  IT 
investment  portfolio(s)  is  important  because  doing  so  provides  the  basis 
for  having  rigor,  discipline,  and  repeatability  in  how  investments  are 
selected  and  controlled  across  the  entire  organization.  Until  DOD  fully 
defines  departmentwide  policies  and  procedures  for  both  individual 
projects  and  portfolios  of  projects,  it  risks  selecting  and  controlling  these 
business  system  investments  in  an  inconsistent,  incomplete,  and  ad  hoc 
manner,  which  in  turn  reduces  the  chances  that  these  investments  will 
meet  mission  needs  in  the  most  cost-effective  manner. 

To  strengthen  DOD’s  business  system  investment  management  capability, 
we  are  recommending  that  the  department  fully  define  the  policies  and 
procedures  associated  with  project-level  and  portfolio-level  investment 
management  as  discussed  in  our  guidance  for  IT  investment  management.8 

In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Business  Transformation)  and  reprinted  in  appendix 
II,  the  department  stated  that  it  agreed  with  the  report’s  overall 
conclusions,  and  it  described  efforts  under  way  and  planned  that  it  said 
would  address  many  of  the  gaps  identified  in  the  report.  In  this  regard,  the 
department  partially  concurred  with  five  of  the  report’s  recommendations, 
adding  that  our  recommendations  and  feedback  are  helpful  in  guiding 
DOD’s  business  transformation  and  related  improvement  efforts. 

However,  the  department  disagreed  with  the  remaining  four 
recommendations  for  two  primary  reasons.  First,  it  stated  that  its  existing 
investment  management  structure  already  satisfies  the  intent  of  these 
recommendations.  For  example,  it  stated  that  its  policies  already  require 
the  provision  of  cost,  schedule,  and  funding  data  as  part  of  investment 
certifications  and  annual  reviews,  and  that  a  linkage  currently  exists 
among  the  investment  selection,  acquisition,  and  funding  processes.  We  do 
not  agree  with  this  reasoning.  Our  recommendations  are  not  intended  to 
address  whether  existing  policies  or  guidance  provide  for  the  use  of  cost, 
schedule,  and  funding  data,  or  whether  they  state  that  investment 
selection,  acquisition,  and  funding  decision  making  are  linked.  Rather,  our 
recommendations  address  the  definitions  of  policy,  guidance,  and 
supporting  procedures  that  fall  short  of  satisfying  the  best  practices 


8GAO-04-394G. 
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embodied  in  our  ITIM  framework.  In  the  case  of  the  above  examples, 
while  we  do  not  question  whether  investment  data  are  provided  to 
investment  decision-making  bodies,  the  department’s  policies  and 
procedures  do  not  include  specific  decision  criteria  that  explain  how  these 
data  are  to  be  used  to  make  consistent,  repeatable  selection  and 
reselection  decisions  across  all  investments.  Furthermore,  while  we  do  not 
question  that  existing  guidance  contains  an  illustration  depicting  a  link 
between  investment  certification  and  review  and  other  DOD  decision 
support  processes,  including  the  funding  process,  neither  this  guidance 
nor  supporting  procedures  define  how  this  linkage  is  executed  (i.e.,  how 
investment  funding  decisions  are  in  fact  integrated  with  investment 
selection  decisions). 

Second,  DOD  stated  that  our  recommendations  contradict  the 
department’s  “tiered  accountability”  approach  to  investment  management, 
in  which  responsibility  and  accountability  for  business  system  investment 
management  is  allocated  between  the  Office  of  the  Secretary  of  Defense 
(corporate  level)  and  DOD  components  (subsidiary  levels)  on  the  basis  of 
investment  size  and  significance.  We  do  not  agree  with  the  department’s 
reasoning.  We  support  DOD’s  tiered  accountability  concept  because  it  is 
consistent  with  the  hierarchical  investment  structures  described  in  our 
ITIM  framework.  Under  the  department’s  current  policies  and  guidance, 
however,  most  DOD  investments  are  not  subject  to  corporate  visibility  and 
oversight,  either  because  they  do  not  involve  development/modemization 
(i.e.,  they  are  in  operations  and  maintenance)  or  because  they  do  not 
exceed  a  certain  dollar  threshold.  Our  framework  recognizes  that  effective 
implementation  of  this  concept  should  include  appropriate  corporate 
visibility  into  and  oversight  of  investments,  either  through  review  and 
approval  of  those  investments  that  meet  certain  criteria  or  through 
awareness  of  a  subordinate  board’s  investment  management  activities. 
Moreover,  this  visibility  and  oversight  should  extend  to  the  entire  portfolio 
of  investments,  including  those  that  are  in  operations  and  maintenance.  To 
ensure  that  this  occurs,  applicable  policies  and  procedures  need  to 
explicitly  cover  all  such  investments  and  need  to  define  how  this  is  to  be 
accomplished. 


Background 


DOD  is  a  massive  and  complex  organization.  To  illustrate,  the  department 
reported  that  its  fiscal  year  2006  operations  involved  approximately  $1.4 
trillion  in  assets  and  $2.0  trillion  in  liabilities,  more  than  2.9  million 
military  and  civilian  personnel,  and  $681  billion  in  net  cost  of  operations. 
To  date,  for  fiscal  year  2007,  the  department  received  appropriations  of 
about  $601  billion.  Organizationally,  the  department  includes  the  Office  of 
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the  Secretary  of  Defense  (OSD),  the  Chairman  of  the  Joint  Chiefs  of  Staff, 
the  military  departments,  numerous  defense  agencies  and  field  activities, 
and  various  unified  combatant  commands  that  are  responsible  for  either 
specific  geographic  regions  or  specific  functions.  (See  fig.  1  for  a 
simplified  depiction  of  DOD’s  organizational  structure.) 


Figure  1:  Simplified  DOD  Organizational  Structure 


Source:  GAO  based  on  DOD  documentation. 


“The  Chairman  of  the  Joint  Chiefs  of  Staff  serves  as  the  spokesman  for  the  commanders  of  the 
combatant  commands,  especially  on  the  administrative  requirements  of  their  commands. 

In  support  of  its  military  operations,  the  department  performs  an 
assortment  of  interrelated  and  interdependent  business  functions, 
including  logistics  management,  procurement,  health  care  management, 
and  financial  management.  As  we  have  previously  reported,9  the  systems 
environment  that  supports  these  business  functions  is  overly  complex  and 
error-prone,  and  is  characterized  by  (1)  little  standardization  across  the 
department,  (2)  multiple  systems  performing  the  same  tasks,  (3)  the  same 
data  stored  in  multiple  systems,  and  (4)  the  need  for  data  to  be  entered 
manually  into  multiple  systems.  Moreover,  according  to  DOD,  this  systems 
environment  is  comprised  of  approximately  3,100  separate  business 
systems.  For  fiscal  year  2007,  Congress  appropriated  approximately  $15.7 
billion  to  DOD,  and  for  fiscal  year  2008,  DOD  has  requested  about  $15.9 
billion  in  appropriated  funds  to  operate,  maintain,  and  modernize  these 
business  systems  and  the  associated  infrastructures. 


9GAO-06-658. 
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As  we  have  previously  reported,* 2 3 * * * * * * 10  the  department’s  nonintegrated  and 
duplicative  systems  impair  DOD’s  ability  to  combat  fraud,  waste,  and 
abuse.  In  fact,  DOD  currently  bears  responsibility,  in  whole  or  in  part,  for 
15  of  our  27  high-risk  areas.11  Eight  of  these  areas  are  specific  to  DOD,12 
and  the  department  shares  responsibility  for  7  other  governmentwide  high- 
risk  areas.13  DOD’s  business  systems  modernization  is  one  of  the  high-risk 
areas,  and  it  is  an  essential  enabler  to  addressing  many  of  the  department’s 
other  high-risk  areas.  For  example,  modernized  business  systems  are 
integral  to  the  department’s  efforts  to  address  its  financial,  supply  chain, 
and  information  security  management  high-risk  areas. 


IT  Investment 
Management  Is  Critical  to 
Achieving  Successful 
Systems  Modernization 


A  corporate  approach  to  IT  investment  management  is  characteristic  of 
successful  public  and  private  organizations.  Recognizing  this,  Congress 
enacted  the  Clinger-Cohen  Act  of  1996, 14  which  requires  the  Office  of 
Management  and  Budget  (OMB)  to  establish  processes  to  analyze,  track, 
and  evaluate  the  risks  and  results  of  major  capital  investments  in  IT 
systems  made  by  executive  agencies.16  In  response  to  the  Clinger-Cohen 
Act  and  other  statutes,  OMB  has  developed  policy  and  issued  guidance  for 
the  planning,  budgeting,  acquisition,  and  management  of  federal  capital 


10See,  for  example,  GAO,  DOD  Travel  Cards:  Control  Weaknesses  Resulted  in  Millions  of 
Dollars  of  Improper  Payments,  GAO-04-576  (Washington,  D.C.:  June  9,  2004);  Military 
Pay:  Army  National  Guard  Personnel  Mobilized  to  Active  Duty  Experienced  Significant 
Pay  Problems,  GAO-04-89  (Washington,  D.C.:  Nov.  13,  2003);  and  Defense  Inventory: 
Opportunities  Exist  to  Improve  Spare  Parts  Support  Aboard  Deployed  Navy  Ships, 
GAO-03-887  (Washington,  D.C.:  Aug.  29,  2003). 

UGAO-07-310. 

12These  8  high-risk  areas  include  DOD’s  (1)  overall  approach  to  business  transfonnation, 

(2)  business  systems  modernization,  (3)  financial  management,  (4)  personnel  security 
clearance  program,  (5)  supply  chain  management,  (6)  support  infrastructure  management, 
(7)  weapon  systems  acquisition,  and  (8)  contract  management. 

13The  7  governmentwide  high-risk  areas  are  (1)  disability  programs,  (2)  ensuring  the 
effective  protection  of  technologies  critical  to  U.S.  national  security  interests, 

(3)  interagency  contracting,  (4)  information  systems  and  critical  infrastructure, 

(5)  information-sharing  for  homeland  security,  (6)  human  capital,  and  (7)  real  property. 

14The  Clinger-Cohen  Act  of  1996,  40  U.S.C.  §§  11101-11704.  This  act  expanded  the 

responsibilities  of  OMB  and  the  agencies  that  had  been  set  under  the  Paperwork  Reduction 

Act  with  regard  to  IT  management.  See  44  U.S.C.  3504(a)(l)(B)(vi)  (OMB);  and  44  U.S.C. 

3506(h)(5)  (agencies). 

15We  have  made  recommendations  to  improve  OMB’s  process  for  monitoring  high-risk  IT 

investments;  see  GAO,  Information  Technology:  OMB  Can  Make  More  Effective  Use  of  Its 

Investment  Reviews,  GAO-05-276  (Washington,  D.C.:  Apr.  15,  2005). 
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IT  Investment  Management:  A 
Brief  Description 


assets.16  We  have  also  issued  guidance  in  this  area,17  which  defines 
institutional  structures,  such  as  the  IRBs;  processes  for  developing 
information  on  investments  (such  as  costs  and  benefits);  and  practices  to 
inform  management  decisions  (such  as  whether  a  given  investment  is 
aligned  with  an  enterprise  architecture). 

IT  investment  management  is  a  process  for  linking  IT  investment  decisions 
to  an  organization’s  strategic  objectives  and  business  plans.  Consistent 
with  this,  the  federal  approach  to  IT  investment  management  focuses  on 
selecting,  controlling,  and  evaluating  investments  in  a  manner  that 
minimize  risks  while  maximizing  the  return  of  investment.18 

During  the  selection  phase,  the  organization  (1)  identifies  and  analyzes 
each  project’s  risks  and  returns  before  committing  significant  funds  to  any 
project  and  (2)  selects  those  IT  projects  that  will  best  support  its  mission 
needs. 

During  the  control  phase,  the  organization  ensures  that  projects,  as  they 
develop  and  investment  expenditures  continue,  meet  mission  needs  at  the 
expected  levels  of  cost  and  risk.  If  the  project  is  not  meeting  expectations 
or  if  problems  arise,  steps  are  quickly  taken  to  address  the  deficiencies. 

During  the  evaluation  phase,  expected  results  are  compared  with  actual 
results  after  a  project  has  been  fully  implemented.  This  comparison  is 
done  to  (1)  assess  the  project’s  impact  on  mission  performance, 

(2)  identify  any  changes  or  modifications  to  the  project  that  may  be 


lbThis  policy  is  set  forth  and  guidance  is  provided  in  OMB  Circular  A-ll  (Nov.  2,  2005) 
(section  300),  and  in  OMB’s  Capital  Programming  Guide ,  which  directs  agencies  to 
develop,  implement,  and  use  a  capital  programming  process  to  build  their  capital  asset 
portfolios. 

17See,  for  example,  GAO-04-394G;  GAO,  Information  Technology:  A  Framework  for 
Assessing  and  Improving  Enterprise  Architecture  Management  ( Version  1.1), 
GAO-03-584G  (Washington,  D.C.:  April  2003);  and  Assessing  Risks  and  Returns:  A  Guide 
for  Evaluating  Federal  Agencies  ’  IT  Investment  Decision-making,  GAO/AIMD-10. 1. 13 
(Washington,  D.C.:  February  1997). 

1SGAO-04-394G;  GAO/AIMD-10. 1.13;  GAO,  Executive  Guide:  Improving  Mission 
Performance  Through  Strategic  Information  Management  and  Technology,  GAO/AIMD- 
94-115  (Washington,  D.C.:  May  1994);  and  Office  of  Management  and  Budget,  Evaluating 
Information  Technology  Investments,  A  Practical  Guide  (Washington,  D.C.:  November 
1995). 
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needed,  and  (3)  revise  the  investment  management  process  based  on 
lessons  learned. 


Overview  of  GAO’s  ITIM  Our  ITIM  framework  consists  of  five  progressive  stages  of  maturity  for  any 
Maturity  Framework  given  agency  relative  to  selecting,  controlling,  and  evaluating  its 

investment  management  capabilities.19  (See  fig.  2  for  the  five  ITIM  stages 
of  maturity.)  This  framework  is  grounded  in  our  research  of  IT  investment 
management  practices  of  leading  private  and  public  sector  organizations. 
The  maturity  stages  are  cumulative;  that  is,  to  attain  a  higher  stage,  an 
agency  must  institutionalize  all  of  the  critical  processes  at  the  lower 
stages,  in  addition  to  the  higher  stage  critical  processes. 

The  framework  can  be  used  to  assess  the  maturity  of  an  agency’s 
investment  management  processes  and  as  a  tool  for  organizational 
improvement.  The  overriding  purpose  of  the  framework  is  to  encourage 
investment  selection  and  control  and  to  evaluate  processes  that  promote 
business  value  and  mission  performance,  reduce  risk,  and  increase 
accountability  and  transparency.  We  have  used  the  framework  in  several 
of  our  evaluations,20  and  a  number  of  agencies  have  adopted  it. 

With  the  exception  of  the  first  stage,  each  maturity  stage  is  composed  of 
“critical  processes”  that  must  be  implemented  and  institutionalized  for  the 
organization  to  achieve  that  stage.  Each  ITIM  critical  process  consists  of 
“key  practices” — to  include  organizational  structures,  policies,  and 
procedures — that  must  be  executed  to  implement  the  critical  process.  It  is 
not  unusual  for  an  organization  to  perform  key  practices  from  more  than 
one  maturity  stage  at  the  same  time.  However,  our  research  shows  that 
agency  efforts  to  improve  investment  management  capabilities  should 


19GAO-04-394G. 

20GAO,  Information  Technology:  Centers  for  Medicare  &  Medicaid  Services  Needs  to 
Establish  Critical  Investment  Management  Capabilities,  GAO-06-12  (Washington,  D.C.: 
Oct.  28,  2005);  Information  Technology:  HHS  Has  Several  Investment  Management 
Capabilities  in  Place,  but  Needs  to  Address  Key  Weaknesses,  GAO-06-11  (Washington, 
D.C.:  Oct.  28,  2005);  Information  Technology:  FAA  Has  Many  Investment  Management 
Capabilities  in  Place,  but  More  Oversight  of  Operational  Systems  Is  Needed,  GAO-04-822 
(Washington,  D.C.:  Aug.  20,  2004);  Bureau  of  Land  Management:  Plan  Needed  to  Sustain 
Progress  in  Establishing  IT  Investment  Management  Capabilities,  GAO-03-1025 
(Washington,  D.C.:  Sept.  12,  2003);  Information  Technology:  Departmental  Leadership 
Crucial  to  Success  of  Investment  Reforms  at  Interior,  GAO-03-1028  (Washington,  D.C.: 
Sept.  12,  2003);  United  States  Postal  Service:  Opportunities  to  Strengthen  IT  Investment 
Management  Capabilities,  GAO-03-3  (Washington,  D.C.:  Oct.  15,  2002);  and  Information 
Technology:  DLA  Needs  to  Strengthen  Its  Investment  Management  Capability, 
GAO-02-314  (Washington,  D.C.:  Mar.  15,  2002). 
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focus  on  implementing  all  lower-stage  practices  before  addressing  higher- 
stage  practices. 

In  the  ITIM  framework,  Stage  2  critical  processes  lay  the  foundation  by 
establishing  successful,  predictable,  and  repeatable  investment  control 
processes  at  the  project  level.  At  this  stage,  the  emphasis  is  on  establishing 
basic  capabilities  for  selecting  new  IT  projects;  controlling  projects  so  that 
they  finish  predictably  within  the  established  cost,  schedule,  and 
performance  expectations;  and  identifying  and  mitigating  exposure  to  risk. 

Stage  3  is  where  the  agency  moves  from  project-centric  processes  to 
portfolio-based  processes  and  evaluates  potential  investments  according 
to  how  well  they  support  the  agency’s  missions,  strategies,  and  goals.  This 
stage  focuses  on  continually  assessing  both  proposed  and  ongoing 
projects  as  part  of  complete  investment  portfolios — integrated  and 
competing  sets  of  investment  options.  It  also  focuses  on  maintaining 
mature,  integrated  selection  (and  reselection);  control;  and 
postimplementation  evaluation  processes.  This  portfolio  perspective 
allows  decision  makers  to  consider  the  interaction  among  investments  and 
the  contributions  to  organizational  mission  goals  and  strategies  that  could 
be  made  by  alternative  portfolio  selections,  rather  than  to  focus 
exclusively  on  the  balance  between  the  costs  and  benefits  of  individual 
investments.  Organizations  implementing  Stages  2  and  3  practices  have  in 
place  capabilities  that  assist  in  establishing  selection,  control,  and 
evaluation  structures,  policies,  procedures,  and  practices  that  are  required 
by  the  investment  management  provisions  of  the  Clinger-Cohen  Act.21 

Stages  4  and  5  require  the  use  of  evaluation  techniques  to  continuously 
improve  both  investment  processes  and  portfolios  to  better  achieve 
strategic  outcomes.  At  Stage  4,  an  organization  has  the  capacity  to  conduct 
IT  succession  activities  and,  therefore,  can  plan  and  implement  the 
deselection  of  obsolete,  high-risk,  or  low-value  IT  investments.  An 
organization  with  Stage  5  maturity  conducts  proactive  monitoring  for 
breakthrough  technologies  that  will  enable  it  to  change  and  improve  its 
business  performance. 


2 ‘The  Clinger-Cohen  Act  of  1996,  40  U.S.C.  §§  11311-11313. 
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Figure  2:  The  Five  ITIM  Stages  of  Maturity  with  Critical  Processes 


X 
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Stage  5:  Leveraging  IT  for 
strategic  outcomes 


-  Optimizing  the  investment  process 

-  Using  IT  to  drive  strategic  business  change 


Stage  4:  Improving  the 
\\  investment  process 

Stage  3:  Developing  a  complete 
investment  portfolio 


-  Improving  the  portfolio's  performance 

-  Managing  the  succession  of  information  systems 

-  Defining  the  portfolio  criteria 

-  Creating  the  portfolio 

-  Evaluating  the  portfolio 

-  Conducting  postimplementation  reviews 


Stage  2:  Building  the  investment 
foundation 
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Stage  1 :  Creating  investment  awareness 


-  Instituting  the  investment  board 

-  Meeting  business  needs 

-  Selecting  an  investment 

-  Providing  investment  oversight 

-  Capturing  investment  information 

IT  spending  without  disciplined  investment  processes 


Source:  GAO. 


DOD’s  major  system  investments  (i.e.,  weapon  and  business  systems)  are 
governed  by  three  management  systems — the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS);  the  Planning,  Programming, 
Budgeting,  and  Execution  (PPBE)  system;  and  the  Defense  Acquisition 
System  (DAS). 

•  JCIDS  is  a  need-driven,  capabilities-based  approach  to  identify  warfighting 
needs  and  meet  future  joint  forces  challenges.  It  is  intended  to  identify 
future  capabilities  for  DOD;  address  capability  gaps  and  mission  needs 
recognized  by  the  Joint  Chiefs  of  Staff  or  derived  from  strategic  guidance, 
such  as  the  National  Security  Strategy  Report22  or  Quadrennial  Defense 
Review;23  and  identify  alternative  solutions  by  considering  a  range  of 
doctrine,  organization,  training,  materiel,  leadership  and  education, 
personnel,  and  facilities  solutions.  According  to  DOD,  the  Joint  Chiefs  of 


Overview  of  DOD’s 
Corporate  Approach  for 
Identifying,  Funding,  and 
Acquiring  All  System 
Investments 


22The  National  Security  Strategy  Report  required  by  50  U.S.C.  404a  is  a  comprehensive 
report  on  the  national  security  strategy  of  the  United  States  submitted  by  the  President  to 
Congress. 

23See  10  U.S.C.  118.  The  Quadrennial  Defense  Review  is  a  comprehensive  examination  of 
the  national  defense  strategy,  force  structure,  force  modernization  plans,  infrastructure, 
budget  plan,  and  other  elements  of  the  defense  program  and  policies  of  the  United  States 
with  a  view  toward  determining  and  expressing  the  defense  strategy  of  the  United  States 
and  establishing  a  defense  program  for  the  next  20  years. 
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Staff,  through  the  Joint  Requirements  Oversight  Council,  has  primary 
responsibility  for  defining  and  implementing  JCIDS. 

•  PPBE  is  a  calendar-driven  approach  that  is  composed  of  four  phases  that 
occur  over  a  moving  2-year  cycle.  The  four  phases — planning, 
programming,  budgeting,  and  executing — define  how  budgets  for  each 
DOD  component  and  the  department  as  a  whole  are  created,  vetted,  and 
executed.  As  recently  reported,24  the  components  start  programming  and 
budgeting  for  addressing  a  JCIDS-identified  capability  gap  or  mission  need 
several  years  before  actual  product  development  under  DAS  begins,  and 
before  OSD  formally  reviews  the  components’  programming  and 
budgeting  proposals  (i.e.,  Program  Objective  Memorandums).  Once 
reviewed  and  approved,  the  financial  details  in  the  Program  Objective 
Memorandums  become  part  of  the  President’s  budget  request  to  Congress. 
During  budget  execution,  components  may  submit  program  change 
proposals  or  budget  change  proposals,  or  both  (e.g.,  program  cost 
increases  or  schedule  delays).  According  to  DOD,  the  OSD  Under 
Secretary  of  Defense  (Policy),  the  Director  for  Program  Analysis  and 
Evaluation,25  and  the  Under  Secretary  of  Defense  (Comptroller)  have 
primary  responsibility  for  defining  and  implementing  the  PPBE  system. 

•  DAS  is  described  in  the  DOD  Directive  5000.1  and  the  DOD  Instruction 
5000. 226  and  establishes  the  procedures  for  the  Defense  Acquisition 
Management  Framework,  which  consists  of  three  event-based  milestones 
associated  with  five  key  program  life-cycle  phases.  These  five  phases  are 
as  follows: 

1.  Concept  Refinement:  Intended  to  refine  the  initial  JCIDS-validated 
system  solution  (concept)  and  create  a  strategy  for  acquiring  the 
investment  solution.  A  decision  is  made  at  the  end  of  this  phase 
(milestone  A  decision)  regarding  whether  to  move  to  the  next  phase 
(Technology  Development). 


24GAO,  Best  Practices:  An  Integrated  Portfolio  Management  Approach  to  Weapon  System 
Investments  Cotdd  Improve  DOD’s  Acquisition  Outcomes,  GAO-07-388  (Washington,  D.C.: 
Mar.  30,  2007). 

25The  Director  for  Program  Analysis  and  Evaluation  is  the  principal  staff  assistant  who 
conducts  independent  analysis  for,  and  provides  independent  advice  on,  all  DOD  program 
and  evaluation  matters  to  the  Secretary  and  Deputy  Secretary  of  Defense. 

26DOD  Directive  S000.1,  May  12,  2003  and  DOD  Instruction  5000.2,  May  12,  2003. 
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2.  Technology  Development:  Intended  to  determine  the  appropriate  set  of 
technologies  to  be  integrated  into  the  investment  solution  by 
iteratively  assessing  the  viability  of  various  technologies  while 
simultaneously  refining  user  requirements.  Once  the  technology  has 
been  demonstrated  in  a  relevant  environment,  a  decision  is  made  at 
the  end  of  this  phase  (milestone  B  decision)  regarding  whether  to 
move  to  the  next  phase  (System  Development  and  Demonstration). 

3.  System  Development  and  Demonstration:  Intended  to  develop  a 
system  or  a  system  increment  and  demonstrate  through  developer 
testing  that  the  system/system  increment  can  function  in  its  target 
environment.  A  decision  is  made  at  the  end  of  this  phase  (milestone  C 
decision)  regarding  whether  to  move  to  the  next  phase  (Production 
and  Deployment). 

4.  Production  and  Deployment:  Intended  to  achieve  an  operational 
capability  that  satisfies  the  mission  needs,  as  verified  through 
independent  operational  test  and  evaluation,  and  ensures  that  the 
system  is  implemented  at  all  applicable  locations. 

5.  Operations  and  Support:  Intended  to  operationally  sustain  the  system 
in  the  most  cost-effective  manner  over  its  life  cycle. 

A  key  principle  of  DAS  is  that  investments  are  assigned  a  category,  where 
programs  of  increasing  dollar  value  and  management  interest  are  subject 
to  more  stringent  oversight.  For  example,  Major  Defense  Acquisition 
Programs  (MDAP)27  and  Major  Automated  Information  Systems  (MAIS)28 
are  large,  expensive  programs  subject  to  the  most  extensive  statutory  and 
regulatory  reporting  requirements  and,  unless  delegated,  are  reviewed  by 
acquisition  boards  at  the  DOD  corporate  level.  Smaller  and  less  risky 
acquisitions  are  generally  reviewed  at  the  component  executive  or  lower 
levels.  Another  key  principle  is  that  DAS  requires  acquisition  management 


2iA  MDAP  is  an  acquisition  program  that  is  estimated  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  to  require  an  eventual  total  expenditure  for 
research,  development,  and  test  and  evaluation  of  more  than  $365  million  (fiscal  year  2000 
constant  dollars)  or,  for  procurement,  of  more  than  $2,190  billion  (fiscal  year  2000  constant 
dollars). 

28A  MAIS  is  a  program  or  initiative  that  is  so  designated  by  the  Assistant  Secretary  of 
Defense  (Networks  and  Information  Integration)/Chief  Information  Officer  or  that  is 
estimated  to  require  program  costs  in  any  single  year  in  excess  of  $32  million  (fiscal  year 
2000  constant  dollars),  total  program  costs  in  excess  of  $126  million  (fiscal  year  2000 
constant  dollars),  or  total  life-cycle  costs  in  excess  of  $378  million  (fiscal  year  2000 
constant  dollars). 
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under  the  direction  of  a  milestone  decision  authority.29  The  milestone 
decision  authority — with  support  from  the  program  manager  and  advisory 
boards,  such  as  the  Defense  Acquisition  Board30  and  the  IT  Acquisition 
Board31 — determines  the  project’s  baseline  cost,  schedule,  and 
performance  commitments.  The  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (USD(AT&L))  has  primary 
responsibility  for  defining  and  implementing  DAS. 


DOD  Business  System 
Investments  Are  Subject  to 
a  Fourth  Management 
System 


DOD’s  business  system  investments  are  also  governed  by  a  fourth 
management  system  that  addresses  how  these  investments  are  reviewed, 
certified,  and  approved  for  compliance  with  the  business  enterprise 
priorities  and  activities  outlined  by  the  business  enterprise  architecture 
(BEA).  For  the  purposes  of  this  report,  we  refer  to  this  fourth  management 
system  as  the  Business  Investment  Management  System.  This  fourth 
management  system  is  described  in  the  following  text  in  terms  of 
governance  entities,  tiered  accountability,  and  business  system  investment 
certification  reviews  and  approvals.  According  to  DOD,  these  four 
management  systems  are  the  means  by  which  DOD  selects,  controls,  and 
evaluates  its  business  system  investments. 


Business  System  Investment  In  2005,  the  department  reassigned  responsibility  for  providing  executive 
Roles  and  Responsibilities  leadership  for  the  direction,  oversight,  and  execution  of  its  business 

systems  modernization  efforts  to  several  entities.  These  entities  and  their 
responsibilities  include  the  following: 


According  to  DOD,  the  milestone  decision  authority  is  the  designated  individual  who  has 
overall  responsibility  for  an  investment.  This  person  has  the  authority  to  approve  an 
investment’s  progression  in  the  acquisition  process  and  is  responsible  for  reporting  cost, 
schedule,  and  performance  results.  For  example,  the  milestone  decision  authority  for  a 
MDAP  program,  when  not  delegated  to  the  component  level,  is  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics,  and  the  milestone  decision  authority 
for  a  MAIS  system  is  the  Assistant  Secretary  of  Defense  (Networks  and  Information 
Integration)/Chief  Information  Officer  or  a  designee. 

30The  Defense  Acquisition  Board,  chaired  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  conducts  reviews  for  MDAPs  at  major  program 
milestones  and  documents  the  decision(s)  resulting  from  the  review  in  an  Acquisition 
Decision  Memorandum. 

31The  IT  Acquisition  Board,  chaired  by  the  Assistant  Secretary  of  Defense  (Networks  and 
Information  Integration)/Chief  Information  Officer,  conducts  reviews  for  MAIS  at  major 
program  milestones  and  documents  the  decision(s)  resulting  from  the  review  in  an 
Acquisition  Decision  Memorandum. 
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•  The  Defense  Business  Systems  Management  Committee  (DBSMC)  serves 
as  the  highest-ranking  governance  body  for  business  systems 
modernization  activities. 

•  The  Principal  Staff  Assistants  serve  as  the  certification  authorities  for 
business  system  modernizations  in  their  respective  core  business 
missions. 

•  The  IRBs  are  chartered  by  the  Principal  Staff  Assistants  and  are  the  review 
and  decision-making  bodies  for  business  system  investments  in  their 
respective  areas  of  responsibility.32 

•  The  component  pre-certification  authority  (PCA)  is  accountable  for  the 
component’s  business  system  investments  and  acts  as  the  component’s 
principal  point  of  contact  for  communication  with  the  IRBs. 

•  The  Business  Transformation  Agency  (BTA)  is  responsible  for  leading  and 
coordinating  business  transformation  efforts  across  the  department.  The 
BTA  is  organized  into  seven  directorates,  one  of  which  is  the  Defense 
Business  Systems  Acquisition  Executive  (DBSAE) — the  component 
acquisition  executive  for  DOD  enterprise-level  (DOD-wide)  business 
systems  and  initiatives.  This  directorate  is  responsible  for  developing, 
coordinating,  and  integrating  enterprise-level  projects,  programs,  systems, 
and  initiatives — including  managing  resources  such  as  fiscal,  personnel, 
and  contracts  for  assigned  systems  and  programs. 

Table  1  lists  these  entities  and  provides  greater  detail  on  their  roles, 
responsibilities,  and  composition.  Figure  3  provides  a  simplified 
illustration  of  the  relationships  among  these  entities. 


32The  four  IRBs  are  for  (1)  Financial  Management,  established  by  the  Deputy  Under 
Secretary  of  Defense  for  Financial  Management;  (2)  Weapon  Systems  Lifecycle 
Management  and  Materiel  Supply  and  Services  Management;  (3)  Real  Property  and 
Installations  Lifecycle  Management,  both  established  by  the  USD(AT&L);  and  (4)  Human 
Resources  Management,  established  by  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness. 


Page  15 


GAO-07-538  Business  Systems  Modernization 


Table  1:  DOD  Business  Investment  Management  System  Entities’  Roles,  Responsibilities,  and  Composition 


Entity 

Roles  and  responsibilities 

DBSMC 

•  Serves  as  approving  authority  for  business  system 
certifications. 

•  Establishes  policies  and  approves  the  business  mission 
area  (BMA)a  strategic  plan,  the  transition  plan  for 
implementation  for  business  systems  modernization,  the 
transformation  program  baseline,  and  the  BEA. 

Support  the  DBSMC’s  management  of  enterprise  business 
IT  investments. 

Serve  as  the  certification  authorities  accountable  for  the 
obligation  of  funds  for  respective  business  systems 
modernization  within  designated  core  business  missions.11 

Provide  the  DBSMC  with  recommendations  for  system 
investment  approval. 

IRBs  •  Serve  as  the  oversight  and  investment  decision-making 

bodies  for  those  business  capabilities  that  support 
activities  under  their  designated  areas  of  responsibility. 

•  Recommend  certification  for  all  business  system 
investments  costing  more  than  $1  million  that  are 
integrated  and  compliant  with  the  BEA. 

Component  PCA  •  Ensures  that  component-level  investment  review 

processes  integrate  with  the  investment  management 
system. 

•  Identifies  those  component  systems  that  require  IRB 
certification  and  prepares,  reviews,  approves,  validates, 
and  transfers  investment  documentation  as  required. 

•  Assesses  and  precertifies  architecture  compliance  of 
component  systems  submitted  for  certification  and  annual 
review. 

•  Acts  as  the  component’s  principal  point  of  contact  for 
communication  with  the  IRBs. 

BTA  •  Serves  as  the  day-to-day  management  entity  of  the 

business  transformation  effort  at  the  DOD  enterprise  level. 

•  Provides  support  to  the  DBSMC  and  the  IRBs. 

•  Operates  under  the  authority  of  the  USD(AT&L)  under  the 
direction  of  the  Deputy  Under  Secretary  of  Defense  for 
Business  Transformation  and  the  Deputy  Under  Secretary 
of  Defense  for  Financial  Management. 


Principal  Staff  Assistants/ 
Certification  Authorities 


Composition 

Chaired  by  the  Deputy  Secretary  of 
Defense;  vice  chair  is  the  USD(AT&L). 
Includes  senior  leadership  in  OSD;  the 
military  departments’  secretaries;  and 
defense  agencies’  heads,  such  as  the 
Assistant  Secretary  of  Defense  (Networks 
and  Information  lntegration)/Chief 
Information  Officer  (ASD(NII)/CIO),  the 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff, 
and  the  commanders  of  the  U.S. 
Transportation  Command  and  the  Joint 
Forces  Command. 

Under  Secretaries  of  Defense  for 
Acquisition,  Technology,  and  Logistics; 
Comptroller;  and  Personnel  and 
Readiness. 


Includes  the  Principal  Staff  Assistants, 
Joint  Staff,  ASD(NII)/CIO,  core  business 
mission  area  representatives,  military 
departments,  defense  agencies,  and 
combatant  commands. 


Includes  the  Chief  Information  Officer  from 
the  Air  Force;  the  Principal  Director  of 
Governance,  Acquisition,  and  Chief 
Knowledge  Office  from  the  Army;  the  Chief 
Information  Officer  from  the  Navy;  and 
comparable  representatives  from  other 
defense  agencies. 


Comprised  of  seven  directorates  (DBSAE, 
Enterprise  Integration,  Transformation 
Planning  and  Performance,  Transformation 
Priorities  and  Requirements,  Investment 
Management,  Warfighter  Support  Office, 
and  Chief  of  Staff). 


Source:  GAO  based  on  DOD  documentation. 
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“According  to  DOD,  the  BMA  is  responsible  for  ensuring  that  capabilities,  resources,  and  materiel  are 
reliably  delivered  to  the  warfighter.  Specifically,  the  BMA  addresses  areas  such  as  real  property  and 
human  resources  management. 

6DOD  has  five  core  business  missions:  Human  Resources  Management,  Weapon  System  Lifecycle 
Management,  Materiel  Supply  and  Services  Management,  Real  Property  and  Installations  Lifecycle 
Management,  and  Financial  Management. 


Figure  3:  Working  Relationships  among  DOD  Business  investment  Management  System  Governance  Entities 


Principal  Staff 
Assistant 
Certification 
Authorities 


IRBs 


Source:  GAO  based  on  DOD  documentation. 


Tiered  Accountability  According  to  DOD,  in  2005  it  adopted  a  tiered  accountability  approach  to 

business  transformation.  Under  this  approach,  responsibility  and 
accountability  for  business  investment  management  is  allocated  between 
the  DOD  corporate  (i.e.,  OSD)  and  the  components  on  the  basis  of  the 
amount  of  development/modemization  funding  involved  and  the 
investment’s  “tier.”  DOD  corporate  is  responsible  for  ensuring  that  all 
business  systems  with  a  development/modernization  investment  in  excess 
of  $1  million  are  reviewed  by  the  IRBs  for  compliance  with  the  BEA, 
certified  by  the  Principal  Staff  Assistants,  and  approved  by  the  DBSMC. 
Components  are  responsible  for  certifying  development/modernization 
investments  with  total  costs  of  $1  million  or  less.  All  DOD  development 
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Business  Investment 
Certification  Reviews  and 
Approvals 


and  modernization  efforts  are  also  assigned  a  tier  on  the  basis  of  the 
acquisition  category  or  the  size  of  the  financial  investment,  or  both. 
According  to  DOD,  a  system  is  given  a  tier  designation  when  it  passes 
through  the  certification  process.  Table  2  describes  the  four  investment 
tiers  and  identifies  the  associated  reviewing  and  approving  entities. 


Table  2:  DOD’s  Investment  Tiers 

Tier  description 

Reviewing/Approving  entities 

Tier  1 

MAIS  and  MDAPs 

IRB  and  DBSMC 

Tier  2 

Exceeding  $10  million  in  total 
development/modernization  costs, 
but  not  designated  MAIS  or  MDAPs 

IRB  and  DBSMC 

Tier  3 

Exceeding  $1  million  and  up  to  $10 
million  in  total 

development/modernization  costs 

IRB  and  DBSMC 

Tier  4 

Investment  funding  required  up  to  $1 
million 

Component-level  review  only  ( unless  the 
system  or  line  of  business  it  supports  is 
designated  as  special  interest  by  the 
Certification  Authority) 

Source:  DOD. 


DOD’s  business  investment  management  system  includes  two  types  of 
reviews  for  business  systems:  certification  and  annual  reviews. 
Certification  reviews  apply  to  new  modernization  projects  with  total  cost 
over  $1  million.  This  review  focuses  on  program  alignment  with  the  BEA 
and  must  be  completed  before  components  obligate  funds  for  programs. 
The  annual  review  applies  to  all  business  programs.  The  focus  for  the 
annual  review  is  to  determine  whether  the  system  development  effort  is 
meeting  its  milestones  and  addressing  its  IRB  certification  conditions. 

Certification  reviews  and  approvals:  Tiers  1  through  3  business  system 
investments  are  certified  at  two  levels — component-level  precertification 
and  corporate-level  certification  and  approval.  At  the  component  level, 
program  managers  prepare,  enter,  maintain,  and  update  information  about 
their  investments  in  the  DOD  IT  Portfolio  Repository  (DITPR),33  such  as 
regulatory  compliance  reporting,  an  architectural  profile,  and 
requirements  for  investment  certification  and  annual  reviews.  The 


33DITPR  is  DOD’s  authoritative  repository  for  certain  information  about  DOD’s  business 
systems,  such  as  system  names  and  the  responsible  DOD  components,  that  are  required  for 
the  certification,  approval,  and  annual  reviews  of  these  business  system  investments. 
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component  PCA  validates  that  the  system  information  is  complete  and 
accessible  on  the  IRB  Portal,  reviews  system  compliance  with  the  BEA 
and  enterprise  transition  plan,  and  verifies  the  economic  viability  analysis. 
The  PCA  asserts  the  status  and  validity  of  the  investment  information  by 
submitting  a  component  precertification  letter  to  the  appropriate  IRB  for 
its  review. 

At  the  corporate  level,  the  IRB  reviews  the  system  information  and 
precertification  letter  submitted  by  the  PCA  to  determine  whether  to 
recommend  investment  certification.  On  completion  of  its  review,  a 
certification  memorandum  is  prepared  and  signed  by  the  designated 
certification  authority34  that  documents  the  IRB’s  system  certification 
decisions  and  any  related  conditions.  The  memorandum  is  then  forwarded 
to  the  DBSMC,  which  either  approves  or  disapproves  the  IRB’s  decisions 
and  issues  a  memorandum  containing  its  decisions.  If  the  DBSMC 
disapproves  a  system  investment,  it  is  up  to  the  component  PCA  to  decide 
whether  to  resubmit  the  investment  after  it  has  resolved  the  relevant 
issues.  Figure  4  provides  a  simplified  overview  of  the  process  flow  of 
certification  reviews  and  approvals. 


34The  certification  authority  is  the  designated  Principal  Staff  Assistant  with  responsibility 
for  review,  approval,  and  oversight  of  the  planning,  design,  acquisition,  deployment, 
operation,  maintenance,  and  modernization  of  defense  business  systems. 
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Figure  4:  Simplified  Process  Flow  of  Certification  Reviews  and  Approvals 
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Source:  GAO  based  on  DOD  documentation. 


Annual  reviews:  Tiers  1  through  4  business  system  investments  are 
annually  reviewed  at  two  levels — the  component  level  and  the  corporate 
level.  At  the  component  level,  program  managers  review  and  update 
information  on  all  tiers  of  investments,  both  in  modernization  and 
operations  and  maintenance,  on  an  annual  basis  in  DITPR.  The  updates  for 
Tiers  1  through  3  with  system  development/modernization  include  cost, 
milestone,  and  risk  variances  and  actions  or  issues  related  to  certification 
conditions.  The  PCA  then  verifies  and  submits  the  information  for  Tiers  1 
through  3  systems  in  development/modernization  for  IRB  review  in  an 
annual  review  assertion  letter.  The  letter  addresses  system  compliance 
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with  the  BEA  and  the  enterprise  transition  plan,  and  includes  investment 
cost,  schedule,  and  performance  information.35 

At  the  corporate  level,  the  IRBs  annually  review  certified  Tiers  1  through  3 
investments  in  development/modernization.  These  reviews  focus  on 
program  compliance  with  the  BEA,  program  performance  against  cost  and 
milestone  baselines,  and  progress  in  meeting  certification  conditions.  The 
IRBs  can  revoke  an  investment’s  certification  when  the  system  has 
significantly  failed  to  achieve  performance  commitments  (i.e.,  capabilities 
and  costs).  When  this  occurs,  the  component  must  address  the  IRB’s 
concerns  and  resubmit  the  investment  for  certification.  Figure  5  shows  a 
simplified  overview  of  the  process  flow  of  annual  reviews. 


Figure  5:  Simplified  Process  Flow  of  Annual  Reviews 


Corporate- 
Level  Annual 
Review 


Component- 
Level  Annual 
Review 


Source:  GAO  based  on  DOD  documentation. 


35In  addition,  each  component  PCA  submits  a  list  of  system  names  to  the  IRBs  on  a 
semiannual  basis,  to  include  Tier  4  systems  and  systems  in  operations  and  maintenance 
that  have  been  reviewed  at  the  component  level. 
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DOD  Has  Established 
the  Structures  Needed 
to  Effectively  Manage 
Business  System 
Investments,  but  Has 
Not  Fully  Defined 
Many  of  the  Related 
Policies  and 
Procedures 


According  to  our  ITIM  framework,  organizations  should  establish  the 
management  structures  needed  to  manage  their  investments  and  build  an 
investment  foundation  by  having  defined  policies  and  procedures  for 
selecting  and  controlling  individual  projects  (Stage  2  capabilities),  and 
organizations  also  should  manage  projects  as  a  portfolio  of  investments 
according  to  defined  policies  and  procedures,  treating  them  as  an 
integrated  package  of  competing  investment  options  and  pursuing  those 
that  best  meet  the  strategic  goals,  objectives,  and  mission  of  the  agency 
(Stage  3  capabilities).  These  Stages  2  and  3  capabilities  assist  agencies  in 
complying  with  the  investment  management  provisions  of  the  Clinger- 
Cohen  Act. 

The  department  has  defined  four  of  nine  practices  that  call  for  project- 
level  policies  and  procedures  (see  table  4)  and  one  of  the  five  practices 
that  call  for  portfolio-level  policies  and  procedures  (see  table  6). 
Specifically,  it  has  established  the  management  structures  contained  in  our 
ITIM  framework,  but  it  has  not  fully  defined  many  of  the  related  policies 
and  procedures. 


With  respect  to  project-level  investment  management  practices,  DOD 
officials  stated  that  these  are  performed  at  the  component  level,  and  that 
departmental  policies  and  procedures  established  for  overseeing 
components’  execution  of  these  practices  are  sufficient.  With  respect  to 
portfolio-level  practices,  however,  these  officials  stated  that  they  intend  to 
improve  departmental  policies  and  procedures  for  business  system 
investments  by,  for  example,  establishing  a  single  governance  structure, 
but  plans  or  time  frames  for  doing  so  have  not  been  established.  According 
to  our  ITIM  framework,  adequately  documenting  both  the  policies  and  the 
associated  procedures  that  govern  how  an  organization  manages  its  IT 
investment  portfolio(s)  is  important  because  doing  so  provides  the  basis 
for  having  rigor,  discipline,  and  repeatability  in  how  investments  are 
selected  and  controlled  across  the  entire  organization.  Until  DOD  fully 
defines  departmentwide  policies  and  procedures  for  both  individual 
projects  and  the  portfolios  of  projects,  it  risks  selecting  and  controlling 
these  business  system  investments  in  an  inconsistent,  incomplete,  and  ad 
hoc  manner,  which  in  turn  reduces  the  chances  that  these  investments  will 
meet  mission  needs  in  the  most  cost-effective  manner. 
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DOD  Has  Begun  to  Build  a 
Foundation  for  Project- 
Level  Investment 
Management,  but  Key 
Policies  and  Procedures 
Are  Not  Fully  Defined 


At  ITIM  Stage  2,  an  organization  has  attained  repeatable  and  successful  IT 
project-level  investment  control  and  basic  selection  processes.  Through 
these  processes,  the  organization  can  identify  project  expectation  gaps 
early  and  take  the  appropriate  steps  to  address  them.  ITIM  Stage  2  critical 
processes  include  (1)  defining  investment  board  operations, 

(2)  identifying  the  business  needs  for  each  investment,  (3)  developing  a 
basic  process  for  selecting  new  proposals  and  reselecting  ongoing 
investments,  (4)  developing  project-level  investment  control  processes, 
and  (5)  collecting  information  about  existing  investments  to  inform 
investment  management  decisions.  Table  3  describes  the  purpose  of  each 
of  these  Stage  2  critical  processes. 


Table  3:  Stage  2  Critical  Processes 

—Building  the  Investment  Foundation 

Critical  process 

Purpose 

Instituting  the  investment  board 

To  define  and  establish  an  appropriate  investment  management  structure  and  the  processes  for 
selecting,  controlling,  and  evaluating  investments. 

Meeting  business  needs 

To  ensure  that  investments  support  the  organization’s  business  needs  and  meet  users’  needs. 

Selecting  an  investment 

To  ensure  that  a  well-defined  and  disciplined  process  is  used  to  select  new  proposals  and 
reselect  ongoing  investments. 

Providing  investment  oversight 

To  review  the  progress  of  investments,  using  predefined  criteria  and  checkpoints,  in  meeting 
cost,  schedule,  risk,  and  benefit  expectations  and  to  take  corrective  action  when  these 
expectations  are  not  being  met. 

Capturing  investment  information 

To  make  available  to  decision  makers  information  to  evaluate  the  impacts  and  opportunities 
created  by  proposed  (or  continuing)  investments. 

Source:  GAO. 


Within  these  five  critical  processes  are  nine  key  practices  that  call  for 
policies  and  procedures  associated  with  effective  project-level 
management.  DOD  has  fully  defined  the  policies  and  procedures  needed  to 
ensure  that  four  of  these  nine  practices  are  performed  in  a  consistent  and 
repeatable  manner.  Specifically,  DOD  has  established  the  management 
structures  by  instituting  an  enterprisewide  investment  board — the 
DBSMC — composed  of  senior  executives,  including  the  Deputy  Secretary 
of  Defense,  with  final  approval  authority  over  associated  subsidiary 
investment  boards.  These  lower-level  investment  boards  include 
representatives  from  combatant  commands,  components,  and  the  Joint 
Chiefs  of  Staff.  In  addition,  DOD’s  business  transformation  and  IRB 
guidance  define  a  process  for  ensuring  that  programs  support  the 
department’s  ongoing  and  future  business  needs.  DOD  also  has  policies 
and  procedures  for  submitting,  updating,  and  maintaining  investment 
information  in  DITPR  and  the  IRB  Portal.  Furthermore,  the  department 
has  assigned  the  component’s  PCA  the  responsibility  to  ensure  that 
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specific  investment  information  contained  in  the  portfolio  repository  and 
the  IRB  Portal  is  accurate  and  complete. 

However,  the  policies  and  procedures  associated  with  the  remaining  five 
project-level  management  practices  are  missing  critical  elements  needed 
to  effectively  carry  out  essential  investment  management  activities.  For 
example: 

•  Policies  and  procedures  for  instituting  the  investment  board  do  not 
address  how  investments  that  are  past  the  development/modernization 
stage  (i.e.,  in  operations  and  maintenance)  are  to  be  governed.  Given  that 
DOD  invests  billions  of  dollars  annually  in  operating  and  maintaining 
business  systems,  this  is  significant.  While  DOD  officials  stated  that 
component-level  policies  and  procedures  address  systems  outside  of 
development/modernization,  our  ITIM  framework  emphasizes  that  the 
corporate  investment  boards  should  continue  to  review  important 
information  about  an  investment,  such  as  cost  and  performance  baselines, 
throughout  the  investment’s  life  cycle.  In  addition,  the  IRB  Concept  of 
Operations  and  other  IRB  documentation  do  not  explicitly  outline  how  the 
business  investment  management  system  is  coordinated  with  JCIDS, 

PPBE,  and  DAS.  Without  clearly  defined  visibility  into  all  investments  with 
an  understanding  of  decisions  reached  through  other  management 
systems,  inconsistent  decisions  may  result. 

•  Procedures  do  not  specify  how  the  full  range  of  cost,  schedule,  and  benefit 
data  is  used  by  the  IRBs  in  making  selection  (i.e.,  certification)  decisions. 
According  to  BTA  officials,  each  IRB  decides  how  to  ensure  compliance 
and  determines  additional  factors  to  consider  when  making  certification 
decisions.  However,  DOD  did  not  provide  us  with  any  supplemental 
policies  or  procedures  for  any  of  the  four  IRBs.  Without  documenting  how 
IRBs  consider  factors  such  as  cost,  schedule,  and  benefits  when  making 
selection  decisions,  the  department  cannot  ensure  that  the  IRBs  and  the 
DBSMC  consistently  and  objectively  select  proposals  that  best  meet  the 
department’s  needs  and  priorities.  Furthermore,  while  the  procedures 
specify  decision  criteria  that  address  statutory  requirements  for  alignment 
to  the  BEA,  the  criteria  allow  programs  to  postpone  demonstrating  full 
compliance  with  several  BEA  artifacts  until  the  final  phases  of  the 
acquisition  process.  As  a  result,  programs  risk  beginning  production  and 
deployment  before  ensuring  that  a  business  system  is  fully  aligned  to  the 
BEA. 

•  Policies  and  procedures  do  not  specify  how  reselection  decisions  at  the 
corporate  level  (i.e.,  annual  review  decisions)  consider  investments  that 
are  in  operations  and  maintenance.  Without  an  understanding  of  how  the 
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IRBs  are  to  consider  these  investments  when  making  reselection 
decisions,  their  ability  to  make  informed  and  consistent  reselection  and 
termination  decisions  is  limited. 

•  Policies  and  procedures  do  not  specify  how  funding  decisions  are 
integrated  with  the  process  of  selecting  an  investment  at  the  corporate 
level.  Without  considering  component  and  corporate  budget  constraints 
and  opportunities,  the  IRBs  risk  making  investment  decisions  that  do  not 
effectively  consider  the  relative  merits  of  various  projects  and  systems 
when  funding  limitations  exist. 

•  Policies  and  procedures  do  not  exist  that  provide  for  sufficient  oversight 
and  visibility  into  component-level  investment  management  activities, 
including  component  reviews  of  systems  in  operations  and  maintenance 
and  Tier  4  investments.  According  to  DOD  officials,  investment  oversight 
is  implemented  through  tiered  accountability,  which,  among  other  things, 
allocates  responsibility  and  accountability  for  business  system 
investments  with  total  costs  of  $1  million  or  less  and  those  in  operations 
and  maintenance  to  the  components.  However,  the  department  did  not 
provide  policies  and  procedures  defining  how  the  DBSMC  and  the  IRBs 
ensure  visibility  into  these  component  processes.  This  is  particularly 
important  because,  according  to  DOD’s  March  15,  2007,  annual  report  to 
Congress,  only  285  of  approximately  3,100  total  business  systems  have 
completed  the  IRB  certification  process  and  have  been  approved  by  the 
DBSMC.  DOD  officials  also  stated  that  the  remaining  business  systems 
have  not  been  through  the  certification  process  and  have  not  been  given  a 
tier  designation.  Without  policies  and  procedures  defining  how  the 
DBSMC  and  the  IRBs  have  visibility  into  and  oversight  of  all  business 
system  investments,  DOD  risks  components  continuing  to  invest  in 
systems  that  are  duplicative,  stovepiped,  nonintegrated,  and  unnecessarily 
costly  to  manage,  maintain,  and  operate. 

Table  4  summarizes  our  findings  relative  to  DOD’s  execution  of  the  nine 
practices  that  call  for  the  policies  and  procedures  needed  to  manage  IT 
investments  at  the  project  level. 
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Table  4:  Summary  of  Policies  and  Procedures  for  Stage  2  Critical  Processes — Building  the  Investment  Foundation 


Critical 

process 

Key  practice 

Rating 

Summary  of  evidence 

Instituting 

the 

investment 

board 

1 .  An  enterprisewide  IT 
investment  board  composed  of 
senior  executives  from  IT  and 
business  units  is  responsible  for 
defining  and  implementing  the 
organization’s  IT  investment 
governance  process. 

Executed 

DOD  has  instituted  an  enterprisewide  business  system  investment  board — 
the  DBSMC — composed  of  senior  executives,  including  the  Deputy 

Secretary  of  Defense  and  the  ASD(NII)/CIO.  This  board  is  responsible  for 
establishing  and  implementing  policies  governing  the  organization’s 
investment  process  and  approving  lower-level  investment  board  processes 
and  procedures. 

2.  The  organization  has  a 
documented  IT  investment 
process  directing  each 
investment  board’s  operations. 

Not 

executed 

DOD’s  IRB  Concept  of  Operations  directs  its  IRBs  and  includes  the  roles 
and  responsibilities  of  the  boards  and  individuals  involved.  However,  the 
concept  of  operations  does  not  assign  the  boards  accountability  for 
programs  throughout  the  investment  life  cycle  (i.e.,  investments  that  are 
past  the  development/modernization  stage  and  in  operations  and 
maintenance).  In  addition,  according  to  our  ITIM  guidance,  the 
department’s  investment  process  should  specify  the  manner  in  which 
investment-related  processes  will  be  coordinated  with  other  organizational 
plans,  processes,  and  documents.  However,  DOD’s  concept  of  operations 
does  not  specify  how  the  business  investment  management  system  is 
coordinated  with  JCIDS,  PPBE,  and  DAS. 

Meeting 

business 

needs 

1 .  The  organization  has 
documented  policies  and 
procedures  for  identifying  IT 
projects  or  systems  that  support 
the  organization’s  ongoing  and 
future  business  needs. 

Executed 

DOD's  Business  Transformation  Guidance  and  the  Investment  Certification 
and  Annual  Review  Process  User  Guidance  define  a  process  for  ensuring 
that  IT  business  system  investments  support  the  department’s  ongoing  and 
future  business  needs. 
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Critical 

process  Key  practice  Rating  Summary  of  evidence 

Not  DOD  has  a  two-stage  selection  process.  The  first  stage  involves  selection 
executed  of  systems  using  the  JCIDS,  DAS,  and  PPBE  management  systems.  At  this 
level,  proposals  and  alternatives  are  viewed  and  prioritized  for  system 
selection.  The  second  stage  of  selection  involves  (1)  certifying  and 
approving  Tiers  1  through  3  investments  and  (2)  elevating  certain 
component  investments  to  an  enterprisewide  status  using  the  business 
investment  management  system. 

While  DOD’s  IRB  Concept  of  Operations  and  its  Investment  Certification 
and  Annual  Review  Process  User  Guidance  define  the  department’s 
corporate  approach  for  certifying  and  approving  investments,  they  do  not 
contain  a  structured  method  defining  how  certification  decisions  are 
reached.  For  example,  the  guidance  does  not  specify  how  cost,  schedule, 
and  benefit  data  are  to  be  used  in  making  certification  decisions.  According 
to  our  ITIM  guidance,  a  structured  selection  method  should  provide 
investment  boards,  business  units,  and  IT  developers  with  a  common 
understanding  of  the  selection  process,  including  the  cost,  schedule,  and 
benefit  data  used  to  compare  and  select  projects.  In  addition,  neither  the 
IRB  Concept  of  Operations  nor  the  Investment  Certification  and  Annual 
Review  Process  User  Guidance  define  the  selection  criteria  used  to  elevate 
these  investments  to  an  enterprisewide  status. 

Furthermore,  the  BEA  Compliance  Guidance  allows  programs  to  postpone 
demonstrating  full  compliance  with  several  BEA  artifacts  until  the  final 
phases  of  the  acquisition  process.  In  addition,  criteria  for  certifying 
compliance  with  the  BEA  are  inconsistently  described  in  DOD 
documentation.  For  example,  the  BEA  Compliance  Guidance  provides 
different  checkpoints  for  assessing  compliance  during  the  life  cycle  of  a 
program  than  the  Business  Transformation  Guidance. 

Not  DOD’s  IRB  Concept  of  Operations  and  the  Investment  Certification  and 
executed  Annual  Review  Process  User  Guidance  define  the  department’s  corporate 
approach  for  annually  reviewing  investments.  However,  these  documents 
do  not  include  specific  criteria  that  describe  how  the  IRBs  make  reselection 
decisions.  For  example,  while  DOD  officials  stated  that  a  program’s  risk 
areas  (i.e.,  cost,  schedule,  and  performance)  are  identified  and  discussed 
by  the  IRB  during  the  annual  reviews,  the  guidance  does  not  specify  how 
this  information  is  used  in  making  annual  review  decisions.  In  addition,  the 
guidance  does  not  provide  for  the  reselection  of  investments  that  are  in 
operations  and  maintenance.  Our  ITIM  guidance  states  that  consistent 
qualitative  and  quantitative  measures  are  needed  for  analyzing  a  project  for 
reselection  or,  if  necessary,  termination.  According  to  ITIM,  the  results  of 
this  analysis  can  help  the  investment  board  determine  the  potential  risk  and 
return  of  continuing  to  fund  an  ongoing  project  and  to  prioritize  projects  on 
the  basis  of  decision  criteria. 

Not  According  to  DOD  officials  and  the  Investment  Certification  and  Annual 
executed  Review  Process  User  Guidance,  the  IRBs  are  aware  of  the  amount  of 

funding  components  have  requested  for  a  program.  However,  this  guidance 
does  not  specify  how  funding  decisions  are  integrated  with  the  process  of 
selecting  an  investment,  and  does  not  specify  how  the  DBSMC  and  the 
IRBs  use  this  information  in  carrying  out  decisions  on  system  certification 
and  approvals. 


2.  The  organization  has 
documented  policies  and 
procedures  for  reselecting 
ongoing  investments. 


3.  The  organization  has 
documented  policies  and 
procedures  for  integrating 
investment  funding  with 
investment  selection. 


Selecting  1 .  The  organization  has 
an  documented  policies  and 

investment  procedures  for  selecting  a  new 
investment. 
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Critical 

process  Key  practice  Rating  Summary  of  evidence 

Providing  1 .  The  organization  has  Not  DOD’s  IRB  Concept  of  Operations  and  the  Investment  Certification  and 

investment  documented  policies  and  executed  Annual  Review  Process  User  Guidance  do  not  provide  sufficient  oversight 

oversight  procedures  for  management  and  visibility  into  component-level  investment  management  activities, 

oversight  of  IT  projects  and  including  component  reviews  of  systems  in  operations  and  maintenance 

systems.  and  Tier  4  investments.  For  example,  while  the  components  submit  a  list  of 

systems  reviewed  at  their  levels,  the  list  lacks  important  project  information, 
including  adherence  to  cost,  schedule,  and  risk  criteria.  According  to  ITIM, 
to  maintain  adequate  oversight,  the  investment  board  should  have  visibility 
into  each  project’s  performance  and  progress  toward  predefined  cost  and 
schedule  expectations  as  well  as  each  project’s  anticipated  benefits  and 
risk  exposure.  In  addition,  IRB  policies  and  procedures  do  not  define  how 
the  department’s  management  systems,  JCIDS,  PPBE,  and  DAS,  are 
related. 


Capturing  1 .  The  organization  has 
investment  documented  policies  and 
information  procedures  for  identifying  and 
collecting  information  about  IT 
projects  and  systems  to  support 
the  investment  management 
process. 


Executed  DOD’s  Investment  Certification  and  Annual  Review  Process  User  Guidance 
describes  the  procedures  for  submitting,  updating,  and  maintaining 
information  in  DITPR  and  the  IRB  Portal,  both  of  which  support  the 
business  investment  management  system. 


2.  An  official  is  assigned 
responsibility  for  ensuring  that  the 
information  collected  during 
project  and  systems  identification 
meets  the  needs  of  the 
investment  management  process. 


Executed  DOD’s  Investment  Certification  and  Annual  Review  Process  User  Guidance 
assigns  the  component  PCA  the  responsibility  to  ensure  investment 
information  contained  in  DITPR  and  the  IRB  Portal  is  accurate  and 
complete.  The  guidance  also  assigns  IRB  staff  responsibility  for  verifying 
these  data. 


Source:  GAO. 


According  to  BTA  officials,  the  IRB  Concept  of  Operations  and  the 
Investment  Certification  and  Annual  Review  Process  User  Guidance  are 
not  intended  to  describe  the  detailed  approach  that  each  IRB  will  use 
when  making  certification  decisions,  adding  that  the  components  are 
responsible  for  selection,  annual  review,  budgeting,  and  acquisition.  While 
the  ITIM  framework  does  allow  for  multiple  entities  to  carry  out 
investment  selection,  control,  and  evaluation,  building  a  sound  investment 
foundation  requires  that  the  enterprisewide  investment  review  board  has 
documented  criteria  and  decision-making  procedures,  clear  integration 
among  investment  decision-support  systems,  and  policies  to  ensure  board 
access  to  system  information  throughout  the  life  cycle  for  all  investments. 
Until  DOD’s  documented  IT  investment  management  policies  and 
procedures  include  fully  defined  policies  and  procedures  for  Stage  2 
activities,  specify  the  linkages  between  the  various  related  processes,  and 
describe  how  investments  are  to  be  governed  in  the  operations  and 
maintenance  phase,  DOD  risks  that  investment  management  activities  will 
not  be  carried  out  consistently  and  in  a  disciplined  manner.  Moreover, 
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DOD  also  risks  selecting  investments  that  will  not  cost-effectively  meet  its 
mission  needs. 


DOD  Has  Assigned 
Responsibility,  but  Has  Not 
Defined  the  Policies  and 
Procedures  Associated 
with  Effective  Portfolio- 
Level  Management 


At  Stage  3,  an  organization  has  defined  critical  processes  for  managing  its 
investments  as  a  portfolio  or  set  of  portfolios.36  Portfolio  management  is  a 
conscious,  continuous,  and  proactive  approach  to  allocating  limited 
resources  among  competing  initiatives  in  light  of  the  investments’  relative 
benefits.  Taking  an  agencywide  perspective  enables  an  organization  to 
consider  its  investments  comprehensively,  so  that  collectively  the 
investments  optimally  address  the  organization’s  missions,  strategic  goals, 
and  objectives.  Managing  IT  investments  as  portfolios  also  allows  an 
organization  to  determine  its  priorities  and  make  decisions  about  which 
projects  to  fund  on  the  basis  of  analyses  of  the  relative  organizational 
value  and  risks  of  all  projects,  including  projects  that  are  proposed,  under 
development,  and  in  operation.  Although  investments  may  initially  be 
organized  into  subordinate  portfolios — on  the  basis  of,  for  example, 
business  lines  or  life-cycle  stages — and  managed  by  subordinate 
investment  boards,  they  should  ultimately  be  aggregated  into  enterprise- 
level  portfolios. 


According  to  ITIM,  Stage  3  involves  (1)  defining  the  portfolio  criteria; 

(2)  creating  the  portfolio;  (3)  evaluating  (i.e.,  overseeing)  the  portfolio;  and 
(4)  conducting  postimplementation  reviews.  Table  5  summarizes  the 
purpose  of  each  of  these  activities. 


Table  5:  Stage  3  Critical  Processes — Developing  a  Complete  Investment  Portfolio 

Critical  process 

Purpose 

Defining  the  portfolio  criteria 

To  ensure  that  the  organization  develops  and  maintains  portfolio  selection  criteria  that 
support  its  mission,  organizational  strategies,  and  business  priorities. 

Creating  the  portfolio 

To  ensure  that  investments  are  analyzed  according  to  the  organization’s  portfolio  selection 
criteria,  and  to  ensure  that  an  optimal  investment  portfolio  with  manageable  risks  and 
returns  is  selected  and  funded. 

Evaluating  the  portfolio 

To  review  the  performance  of  the  organization’s  investment  portfolio(s)  at  agreed-upon 
intervals,  and  to  adjust  the  allocation  of  resources  among  investments  as  necessary. 

Conducting  postimplementation  reviews 

To  compare  the  results  of  recently  implemented  investments  with  the  expectations  that 
were  set  for  them,  and  to  develop  a  set  of  lessons  learned  from  these  reviews. 

Source:  GAO. 


36Investment  portfolios  are  integrated  agencywide  collections  of  investments  that  are 
assessed  and  managed  collectively  on  the  basis  of  common  criteria. 
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DOD  is  executing  one  of  the  five  practices  within  these  four  critical 
processes  that  call  for  policies  and  procedures  associated  with  effective 
portfolio-level  management.  Specifically,  DOD  has  issued  departmentwide 
guidance37  that  assigns  responsibilities  to  the  USD(AT&L)  for  managing 
and  establishing  business  system  investment  portfolios,  including 
leveraging  or  establishing  a  governance  forum  to  oversee  these  business 
system  investment  portfolio  activities. 

However,  DOD  has  not  fully  defined  the  policies  and  procedures  needed  to 
effectively  execute  the  remaining  four  portfolio  management  practices 
relative  to  business  system  investments.  Specifically,  DOD  does  not  have 
policies  and  procedures  for  defining  the  portfolio  criteria  or  for  creating 
and  evaluating  the  portfolio.  In  addition,  while  DOD  has  policies  and 
procedures  for  conducting  postimplementation  reviews  as  part  of  DAS, 
these  reviews  do  not  address  systems  at  all  tier  levels.  Furthermore,  there 
are  no  procedures  detailing  how  lessons  learned  from  these  reviews  are 
used  during  investment  review  as  the  basis  for  management  and  process 
improvements. 

Table  6  summarizes  the  rating  for  each  critical  process  required  to  manage 
investment  as  a  portfolio  and  summarizes  the  evidence  that  supports  these 
ratings. 


3iDOD  Directive  8115.01,  Information  Technology  Portfolio  Management,  and  DOD 
Instruction  8115.02,  Information  Technology  Portfolio  Management  Implementation. 
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Table  6:  Summary  of  Policies  and  Procedures  for  Stage  3  Critical  Processes — Developing  a  Complete  Investment  Portfolio 

Critical  process 

Key  practice 

Rating 

Summary  of  evidence 

Defining  the  portfolio 
criteria 

1.  The  organization  has  documented 
policies  and  procedures  for  creating 
and  modifying  IT  portfolio  selection 
criteria. 

Not 

executed 

DOD’s  IT  Portfolio  Management  Implementation  states 
that  the  USD(AT&L)  is  responsible  for  creating  and 
modifying  portfolio  criteria  (e.g.,  prioritization  and 
investment  tradeoffs)  for  business  system  investments. 
However,  the  USD(AT&L)  has  not  documented  the 
related  policies  and  procedures. 

2.  Responsibility  is  assigned  to  an 
individual  or  group  for  managing  the 
development  and  modification  of  the 
IT  portfolio  selection  criteria. 

Executed 

DOD’s  IT  Portfolio  Management  assigns  responsibility 
for  the  business  mission  area  portfolio  management  to 
the  USD(AT&L),  who  leads  and  manages  business 
system  investments  in  coordination  with  the 
ASD(NII)/CIO,  the  Under  Secretary  of  Defense 
(Comptroller),  and  the  Under  Secretary  of  Defense 
(Personnel  and  Readiness). 

Creating  the  portfolio 

1 .  The  organization  has  documented 
policies  and  procedures  for 
analyzing,  selecting,  and  maintaining 
the  investment  portfolios. 

Not 

executed 

DOD  does  not  have  policies  and  procedures  for 
analyzing,  selecting,  and  maintaining  business  system 
investment  portfolios. 

Evaluating  the  portfolio 

1 .  The  organization  has  documented 
policies  and  procedures  for 
reviewing,  evaluating,  and  improving 
the  performance  of  its  portfolio(s). 

Not 

executed 

While  the  IRB  Concept  of  Operations  states  that  the 

IRBs  are  responsible  for  reviewing  factors  associated 
with  portfolio  management,  such  as  architecture 
alignment  and  capability  delivery,  there  are  no  policies 
and  procedures  indicating  how  the  IRBs  should  use 
these  factors  and  project  indicators — such  as  cost, 
schedule,  and  risk — to  review,  evaluate,  and  improve 
their  portfolios.  According  to  our  HIM  guidance  for 

Stage  3,  IRBs  should  use  actual  investment  data,  such 
as  project  cost  and  adherence  to  schedule,  as  the  basis 
for  reviewing  and  evaluating  its  portfolio(s)  to  ensure 
that  the  overall  portfolio  provides  the  maximum  benefits 
at  a  desired  cost  and  at  an  acceptable  level  of  risk. 

Conducting 

postimplementation 

reviews 

1.  The  organization  has  documented 
policies  and  procedures  for 
conducting  postimplementation 
reviews. 

Not 

executed 

While  DOD  requires  postimplementation  reviews  for 

Tier  1  systems  as  part  of  DAS,  there  are  no  policies  or 
procedures  for  conducting  them  for  Tiers  2  or  3 
systems.  Moreover,  there  are  no  policies  or  procedures 
directing  the  DBSMC  or  IRBs,  or  both,  which  are 
accountable  for  corporate  business  system 
investments,  to  consider  information  gathered  and  to 
develop  lessons  learned  from  these  postimplementation 
reviews.  According  to  HIM,  an  effective 
postimplementation  review  includes,  among  other 
things,  how  conclusions,  lessons  learned,  and 
recommended  management  action  steps  are  to  be 
disseminated  to  executives  and  others. 

Source:  GAO. 


According  to  BTA  officials,  while  portfolio  management  is  primarily  a 
component  responsibility,  they  are  working  toward  developing  more 
effective  departmentwide  portfolio  management  processes,  but  plans  or 
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time  frames  for  doing  so  have  not  been  established.  Without  defining 
corporate  policies  and  procedures  for  managing  business  system 
investment  portfolios,  DOD  is  at  risk  of  not  consistently  selecting  the  mix 
of  investments  that  best  supports  the  departmentwide  mission  needs  and 
ensuring  that  investment-related  lessons  learned  are  shared  and  applied 
departmentwide. 


Conclusions 


Given  the  importance  of  business  systems  modernization  to  DOD’s 
mission,  performance,  and  outcomes,  it  is  vital  for  the  department  to  adopt 
and  employ  an  effective  institutional  approach  to  managing  business 
system  investments.  While  the  department  has  established  aspects  of  such 
an  approach  and,  thus,  has  a  foundation  on  which  to  build,  it  is  lacking 
other  important  elements,  such  as  specific  policies  and  procedures  needed 
for  project-level  and  portfolio-level  investment  management,  including 
integration  with  DOD’s  other  key  management  systems  and  sufficient 
oversight  and  visibility  into  operations  and  maintenance  investments  and 
Tier  4  investments.  This  means  that  DOD  lacks  an  institutional  capability 
to  ensure  that  it  is  investing  in  business  systems  that  best  support  its 
strategic  needs,  and  that  ongoing  projects  meet  cost,  schedule,  and 
performance  expectations.  Until  DOD  develops  this  capability,  the 
department  will  be  impaired  in  its  ability  to  optimize  business  mission  area 
performance  and  accountability. 


Recommendations  for  strengthen  DOD’s  business  system  investment  management  capability 

and  address  the  weaknesses  discussed  in  this  report,  we  recommend  that 
Executive  Action  the  Secretary  of  Defense  direct  the  Deputy  Secretary  of  Defense,  as  the 

chair  of  the  DBSMC,  to  ensure  that  well-defined  and  disciplined  business 
system  investment  management  policies  and  procedures  are  developed 
and  issued.  At  a  minimum,  this  should  include  project-level  management 
policies  and  procedures  that  address  the  following  five  areas: 

•  instituting  the  investment  boards,  including  assigning  the  investment 
boards  responsibility,  authority,  and  accountability  for  programs 
throughout  the  investment  life  cycle  and  specifying  how  the  business 
investment  management  system  is  coordinated  with  JCIDS,  PPBE,  and 
DAS; 
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•  selecting  new  investments,  including  specifying  how  cost,  schedule,  and 
benefit  data  are  to  be  used  in  making  certification  decisions;  defining  the 
criteria  used  to  select  investments  as  enterprisewide;  and  establishing 
consistent  and  effective  guidance  for  BEA  compliance; 

•  reselecting  ongoing  investments,  including  specifying  how  cost,  schedule, 
and  performance  data  are  to  be  used  in  the  annual  review  process  and 
providing  for  the  reselection  of  investments  that  are  in  operations  and 
maintenance; 

•  integrating  funding  with  the  process  of  selecting  an  investment,  including 
specifying  how  the  DBSMC  and  the  IRBs  use  funding  information  in 
carrying  out  decisions  on  system  certification  and  approvals;  and 

•  overseeing  IT  projects  and  systems,  including  providing  sufficient 
oversight  and  visibility  into  component-level  investment  management 
activities. 

These  well-defined  and  disciplined  business  system  investment 
management  policies  and  procedures  should  also  include  portfolio-level 
management  policies  and  procedures  that  address  the  following  four 
areas: 

•  creating  and  modifying  IT  portfolio  selection  criteria  for  business  system 
investments; 

•  analyzing,  selecting,  and  maintaining  business  system  investment 
portfolios; 

•  reviewing,  evaluating,  and  improving  the  performance  of  its  portfolio(s)  by 
using  project  indicators,  such  as  cost,  schedule,  and  risk;  and 

•  conducting  postimplementation  reviews  for  all  investment  tiers  and 
directing  the  investment  boards,  which  are  accountable  for  corporate 
business  system  investments,  to  consider  the  information  gathered  and  to 
develop  lessons  learned  from  these  reviews. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Business  Transformation)  and  reprinted  in  appendix 
II,  the  department  stated  that  it  agreed  with  the  report’s  overall 
conclusions,  and  it  described  efforts  under  way  and  planned  that  it  said 
would  address  many  of  the  gaps  identified  in  the  report.  In  this  regard,  the 
department  partially  concurred  with  five  of  the  report’s  recommendations, 
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adding  that  our  recommendations  and  feedback  are  helpful  in  guiding 
DOD’s  business  transformation  and  related  improvement  efforts. 
Nevertheless,  the  department  disagreed  with  the  remaining  four 
recommendations  on  the  grounds  that  their  intent  had  already  been  met 
through  DOD’s  existing  business  system  investment  management 
structure  and  processes,  or  that  they  contradicted  the  tiered  accountability 
concept  embedded  in  this  structure  and  processes.  The  department’s 
comments  relative  to  each  of  our  project-level  and  portfolio-level 
recommendations,  along  with  our  responses  to  its  comments,  are  provided 
below. 

With  respect  to  our  five  project-level  recommendations,  the  department 
stated  that  it  partially  agreed  with  two  and  disagreed  with  three. 

•  DOD  partially  agreed  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  assign  the  investment  boards  responsibility 
for  programs  throughout  the  investment  life  cycle  and  specify  how  the 
business  investment  management  system  is  coordinated  with  JCIDS, 

PPBE,  and  DAS.  In  particular,  it  stated  that  under  its  tiered  accountability 
approach  to  business  systems  investment  management,  the  components 
are  currently  required  to  review  all  programs  throughout  their  investment 
life  cycles.  We  do  not  question  this  requirement,  and  we  recognize  it  in  our 
report.  However,  consistent  with  our  ITIM  framework,  the  corporate 
investment  boards  should  continue  to  review  investments  that  meet  the 
defined  threshold  criteria  throughout  their  life  cycles  (i.e.,  when  they  are 
in  operations  and  maintenance).  In  contrast,  DOD’s  corporate  boards 
focus  only  on  those  investments  that  are  in  the 
development/modernization  stage.  The  department  also  stated  that  a 
linkage  is  currently  depicted  in  existing  guidance  among  its  investment 
selection,  acquisition,  and  funding  processes.  While  we  do  not  question 
that  this  guidance  contains  an  illustration  depicting  such  a  link,  neither 
this  guidance  nor  supporting  procedures  define  how  this  linkage  is 
executed  (e.g.,  how  investment  funding  decisions  are  in  fact  integrated 
with  investment  selection  decisions).  DOD’s  comments  appear  to 
acknowledge  this  point  by  stating  that  the  department  has  begun  to  define 
and  implement  a  Business  Capability  Lifecycle  concept,  which  is  intended 
to  integrate  the  investment  selection  and  acquisition  management 
processes  for  Tier  1  and  enterprise  systems  into  a  single  oversight  process 
that  leverages  the  existing  IRB  and  DBSMC  oversight  framework. 

•  DOD  partially  agreed  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  specify  how  cost,  schedule,  and  benefit  data 
are  to  be  used  in  making  certification  and  annual  review  decisions;  define 
the  criteria  used  to  select  investments  as  enterprisewide;  and  establish 
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consistent  and  effective  guidance  for  BEA  compliance.  In  particular,  the 
department  agreed  that  additional  criteria  are  required  for  selecting 
enterprisewide  investments,  noting  that  initial  criteria  have  been  defined 
and  will  be  incorporated  in  the  investment  management  process.  However, 
the  department  did  not  agree  that  cost,  schedule,  and  BEA  compliance 
information  are  not  sufficiently  used  for  certification  and  annual  review 
decisions,  adding  that  such  information  is  required  in  its  current  policies. 
We  do  not  agree.  Specifically,  while  we  do  not  question  whether 
investment  data  are  provided  to  the  DBSMC  and  the  IRBs,  the 
department’s  policies  and  procedures  do  not  include  specific  decision 
criteria  that  explain  how  these  data  are  to  be  used  to  make  consistent, 
repeatable  selection  and  reselection  decisions  across  all  investments.  In 
addition,  while  BEA  compliance  policies  have  been  developed  and  are 
being  used,  the  guidance  is  not  fully  defined.  For  example,  the  guidance 
allows  programs  to  defer  demonstrating  full  compliance  with  important 
BEA  artifacts  until  the  final  phases  of  the  acquisition  process,  at  which 
time  addressing  instances  of  noncompliance  would  be  more  expensive  and 
difficult.  Furthermore,  the  compliance  criteria  are  not  consistently 
described  in  different  guidance  documentation.  As  a  result,  DOD  risks 
beginning  system  production  and  deployment  before  ensuring  that  a 
system  is  sufficiently  aligned  to  the  BEA. 

•  DOD  did  not  agree  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  provide  for  the  reselection  of  investments 
that  are  in  operations  and  maintenance.  According  to  DOD,  components 
are  required  by  policy  to  annually  review  all  business  systems,  including 
investments  for  which  there  is  no  planned  development  or  modernization 
spending.  We  agree  that  the  annual  review  process  does  require  this. 
However,  consistent  with  our  ITIM  framework,  the  corporate  investment 
boards  should  continue  to  reselect  investments  that  meet  the  defined 
threshold  criteria  throughout  their  life  cycles  (i.e.,  when  they  are  in 
operations  and  maintenance).  In  contrast,  DOD’s  corporate  boards  focus 
only  on  reselecting  those  investments  that  are  in  the 
development/modernization  stage. 

•  DOD  did  not  agree  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  specify  how  the  corporate  boards  use  funding 
information  in  carrying  out  decisions  on  system  certification  and 
approvals.  In  this  regard,  it  stated  that  such  information  is  required  in  its 
current  policies  and  considered  during  board  deliberations.  We  do  not 
agree.  Our  recommendation  does  not  address  whether  existing  policies  or 
guidance  provide  for  the  collection  of  this  information;  our 
recommendation  addresses  the  definition  of  policy,  guidance,  and 
supporting  procedures  that  fall  short  of  satisfying  the  best  practices 
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embodied  in  our  ITIM  framework.  Specifically,  while  we  do  not  question 
whether  funding  data  are  provided  to  investment  decision-making  bodies, 
the  department’s  policies  and  procedures  do  not  include  specific  decision 
criteria  that  explain  how  these  data  are  to  be  used  to  make  consistent, 
repeatable  selection  and  reselection  decisions  across  all  investments. 

•  DOD  did  not  agree  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  provide  for  sufficient  oversight  and  visibility 
into  component-level  investment  management  activities.  In  particular,  it 
stated  that  this  recommendation  contradicts  the  department’s  “tiered 
accountability”  approach  to  investment  management.  We  do  not  agree. 
Under  the  department’s  current  policies  and  guidance,  most  DOD 
investments  are  not  subject  to  corporate  visibility  and  oversight,  either 
because  they  do  not  involve  development/modernization  (i.e.,  they  are  in 
operations  and  maintenance)  or  because  they  do  not  exceed  a  certain 
dollar  threshold.  Our  framework  recognizes  that  effective  implementation 
of  a  tiered  accountability  concept  should  include  appropriate  corporate 
visibility  into  and  oversight  of  investments,  either  through  review  and 
approval  of  those  investments  that  meet  certain  criteria  or  through 
awareness  of  a  subordinate  board’s  investment  management  activities. 
Moreover,  this  visibility  and  oversight  should  extend  to  the  entire  portfolio 
of  investments,  including  those  that  are  in  operations  and  maintenance.  To 
ensure  that  this  occurs,  applicable  policies  and  procedures  need  to 
explicitly  cover  all  such  investments  and  need  to  define  how  this  is  to  be 
accomplished. 

With  respect  to  our  four  portfolio-level  recommendations,  the  department 
stated  that  it  partially  agreed  with  three  and  disagreed  with  one. 

•  DOD  partially  agreed  with  our  recommendation  to  define  and  implement 
policies  and  procedures  for  creating  and  modifying  portfolio  selection 
criteria  for  business  system  investments.  In  particular,  it  stated  that  while 
components  are  responsible  for  developing  and  managing  their  own 
portfolio  management  processes,  upcoming  initiatives,  such  as  the 
Business  Capability  Lifecycle  concept,  will  lead  to  revisions  in  the 
department’s  investment  review  policies  and  procedures,  such  as 
including  portfolio  selection  criteria  for  enterprise  systems  that  span 
components.  However,  while  these  are  important  steps,  the  concept,  as 
defined  by  the  department,  does  not  apply  to  the  thousands  of  investments 
that  are  not  enterprisewide. 

•  DOD  partially  agreed  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  address  analyzing,  selecting,  and  maintaining 
business  system  investment  portfolios.  In  particular,  it  stated  that  the 
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implementation  of  the  Business  Capability  Lifecyle  concept  will  provide 
the  corporate  boards  with  improved  visibility  into  all  investments  in  a 
given  portfolio  and  a  broader  set  of  criteria  for  analyzing,  selecting,  and 
maintaining  business  system  investment  portfolios. 

•  DOD  partially  agreed  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  address  reviewing,  evaluating,  and  improving 
the  performance  of  its  portfolio(s)  by  using  cost,  schedule,  and  risk 
indicators.  In  particular,  it  stated  that  while  such  indicators  are  part  of  the 
investment  certification  and  review  processes,  efforts  are  now  under  way 
to  better  understand  the  nature  and  impact  of  program  risks  through 
application  of  an  Enterprise  Risk  Assessment  Methodology.  While  we 
recognize  the  role  and  value  of  such  tools  in  understanding  and  addressing 
program  risks,  this  tool  is  program-specific  and  not  portfolio-focused. 

•  DOD  did  not  agree  with  our  recommendation  to  define  and  implement 
policies  and  procedures  that  address  conducting  postimplementation 
reviews  and  having  the  corporate  investment  boards  consider  the  review 
results  and  develop  lessons  learned  from  them.  In  particular,  it  stated  that 
this  process  should  not  be  managed  by  the  Deputy  Secretary  of  Defense, 
and  also  stated  that  our  recommendation  is  redundant  with 
postimplementation  reviews  currently  required  under  OMB  Circular  A- 
130.38  We  do  not  agree  with  DOD’s  statements.  Our  recommendation  does 
not  call  for  the  Deputy  Secretary  to  manage  the  postimplementation 
review  process.  Rather,  it  provides  for  developing  policies  and  procedures 
for  performing  postimplementation  reviews  for  all  tiers  of  business 
systems  and  having  the  DBSMC  and  IRBs,  which  are  the  corporate 
investment  boards,  consider  the  information  gathered  from  these  reviews 
and  develop  lessons  learned. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees;  the  Director,  Office  of  Management  and  Budget;  the  Secretary 
of  Defense;  the  Deputy  Secretary  of  Defense;  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics;  the  Under  Secretary  of 
Defense  (Comptroller);  the  Assistant  Secretary  of  Defense  (Networks  and 
Information  Integration)/Chief  Information  Officer;  the  Under  Secretary  of 


38 According  to  OMB  Circular  A-130,  which  establishes  policy  for  the  management  of  federal 
information  resources,  as  part  of  the  capital  planning  process,  an  agency  must,  among 
other  things,  conduct  postimplementation  reviews  of  information  systems  and  information 
resource  management  processes  to  validate  estimated  benefits  and  costs;  document 
effective  management  practices  for  broader  use;  and  document  lessons  learned  from  the 
postimplementation  reviews. 
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Defense  (Personnel  and  Readiness);  and  the  Director,  Defense  Finance 
and  Accounting  Service.  Copies  of  this  report  will  be  made  available  to 
other  interested  parties  upon  request.  This  report  will  also  be  available  at 
no  charge  on  our  Web  site  at  http://www.gao.gov. 

If  you  or  your  staffs  have  any  questions  on  matters  discussed  in  this 
report,  please  contact  me  at  (202)  512-3439  or  hiter@gao.gov.  Contact 
points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may 
be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  major 
contributions  to  this  report  are  listed  in  appendix  III. 


Randolph  C.  Hite 

Director,  Information  Technology  Architecture 
and  Systems  Issues 
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List  of  Committees 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Daniel  Inouye 
Chairman 

The  Honorable  Ted  Stevens 
Ranking  Member 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Ike  Skelton 
Chairman 

The  Honorable  Duncan  Hunter 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  John  P.  Murtha 
Chairman 

The  Honorable  C.W.  Bill  Young 
Ranking  Member 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Objective,  Scope,  and 
Methodology 


Our  objective  was  to  determine  whether  the  Department  of  Defense’s 
(DOD)  corporate  investment  management  approach  comports  with 
relevant  federal  guidance.  Our  analysis  was  based  on  the  best  practices 
contained  in  GAO’s  Information  Technology  Investment  Management 
(ITIM)  framework,  and  the  framework’s  associated  evaluation 
methodology,  and  focused  on  DOD’s  establishment  of  departmental-level 
policies  and  procedures  for  business  system  investments  needed  to  assist 
organizations  in  complying  with  the  investment  management  provisions  of 
the  Clinger-Cohen  Act  of  1996  (Stages  2  and  3).  It  did  not  include  case 
studies  to  verify  the  implementation  of  established  policies  and 
procedures. 

To  address  our  objective,  we  asked  DOD  to  complete  a  self-assessment  of 
its  corporate  investment  management  process  and  provide  the  supporting 
documentation.  We  then  reviewed  the  results  of  the  department’s  self- 
assessment  of  Stages  2  and  3  organizational  commitment  practices — 
meaning  those  practices  related  to  structures,  policies,  and  procedures — 
and  compared  them  against  our  ITIM  framework.  We  also  validated  and 
updated  the  results  of  the  self-assessment  through  document  reviews  and 
interviews  with  officials,  such  as  the  Director  of  Investment  Management 
and  the  Defense  Business  Systems  Acquisition  Executive.  In  doing  so,  we 
reviewed  written  policies,  procedures,  and  guidance  and  other 
documentation  providing  evidence  of  executed  practices,  including  the 
Defense  Acquisition  System  guidance,  the  Investment  Review  Board  (IRB) 
Concept  of  Operations  and  Guidance,  the  Business  Enterprise 
Architecture  Compliance  Guidance,  IRB  charters  and  meeting  minutes, 
and  the  Business  Transformation  Guidance. 

We  compared  the  evidence  collected  from  our  document  reviews  and 
interviews  with  the  key  practices  in  ITIM.  We  rated  the  key  practices  as 
“executed”  on  the  basis  of  whether  the  agency  demonstrated  (by  providing 
evidence  of  performance)  that  it  had  met  all  of  the  criteria  of  the  key 
practice.  A  key  practice  was  rated  as  “not  executed”  when  we  found 
insufficient  evidence  of  all  elements  of  a  practice  being  fully  performed  or 
when  we  determined  that  there  were  significant  weaknesses  in  DOD’s 
execution  of  the  key  practice.  In  addition,  we  provided  DOD  with  the 
opportunity  to  produce  evidence  for  the  key  practices  rated  as  “not 
executed.” 

We  conducted  our  work  at  DOD  headquarters  offices  in  Arlington, 

Virginia,  from  August  2006  through  April  2007  in  accordance  with 
generally  accepted  government  auditing  standards. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


ACQUISITION. 
TECHNOLOGY 
AND  LOGISTICS 


MAY  3  2007 


Mr.  Randolph  Hite 

Director,  Information  Technology  Architecture  and  Systems  Issues 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 


Dear  Mr.  Hite: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  07- 
538,  “BUSINESS  SYSTEMS  MODERNIZATION:  DoD  Needs  to  Fully  Define  Policies 
and  Procedures  for  Institutionally  Managing  Investments,”  dated  March  30,  2007,  (GAO 
Code  310636). 

The  Department  welcomes  GAO's  insight  and  suggestions  as  we  continue  to  strive 
toward  meeting  our  shared  goals  of  transforming  defense  business  practices.  GAO 
provides  valuable  feedback  on  the  Department’s  achievements,  highlights  areas  where  we 
can  improve,  and  helps  keep  our  effort  on  track  toward  achieving  quality  outcomes. 

Attached  are  the  Department’s  responses  to  the  GAO’s  recommendations  to  draft 
report  GAO-07-538.  The  Department  partially  concurs  on  five  and  non-concurs  with  four 
of  the  recommendations  because  we  believe  that  the  existing  structure  established  by  the 
Department  already  meets  the  overall  intent  of  several  of  GAO’s  recommendations. 

However,  we  agree  with  GAO’s  overall  conclusions  that  DoD  should  continue  to 
improve  upon  its  existing  investment  management  policies  and  procedures  for  individual 
business  systems  and  programs.  In  fact,  the  Department  is  now  developing  and 
implementing  changes  in  its  investment  management  practices  that  address  many  of  the 
gaps  identified  by  GAO  in  this  audit  report.  These  efforts,  in  the  totality,  address  many  of 
the  issues  and  illustrate  preplanned  BTA  efforts  to  ameliorate  the  concerns.  Recent 
enterprise-level  improvements  include: 

•  Risk  mitigation.  Five  of  the  ten  business  enterprise-level  business  programs 
defined  as  Major  Automated  Information  Systems  (MAIS)  have  been  or  are 
scheduled  soon  for  an  Enterprise  Risk  Assessment  Methodology  (ERAM) 
evaluation  of  execution  risk  and  alignment  with  enterprise  capability  goals.  The 
remainder  of  these  10  business  MAIS  will  be  brought  under  ERAM  by  the  end  of 
FY  2007. 

•  Enterprise  standards.  The  BTA  is  currently  “rationalizing  the  enterprise”  and 
identifying  systems  as  “enterprise”  or  “non-enterprise”.  Following  the  initial 
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declaration,  the  programs  assigned  to  the  “enterprise”  will  be  under  the  direction 
of  Defense  Business  Systems  Acquisition  Executive  (DBSAE)  and  “non¬ 
enterprise”  programs  will  be  further  assigned  to  the  appropriate  component,  thus 
examining  and  assigning  the  programs  to  comport  with  the  DoD  tiered 
accountability  structure.  While  this  effort  is  in  its  infancy,  it  provides  increased 
insight  into  programs,  and  the  appropriate  level  of  portfolio  management. 

•  Management  framework.  We  are  developing  specific  policy  guidance  to  amend 
the  non-statutory  portions  of  the  DoD  5000  series  of  acquisition  regulations  and 
the  JCS  3 1 70  to  adopt  a  management  structure  tailored  to  the  business  mission 
area.  This  framework,  called  the  Business  Capability  Lifecycle  (BCL),  is 
beginning  implementation.  BCL  is  being  designed  to  directly  address 
acknowledged  shortfalls  in  how  DoD  develops  and  fields  MAIS  and  enterprise- 
level  business  systems.  We  expect  to  fully  implement  BCL  early  in  FY  2008. 

At  the  component  level,  the  tiered  accountability  concept  remains  the  foundation  for 
implementing  portfolio  management  for  the  business  mission  area.  Although  we  agree 
that  at  an  enterprise  level  we  need  to  establish  the  appropriate  guidance  and  infrastructure 
for  business  transformation,  we  strongly  believe  that  delegating  certain  investment 
management  responsibilities  to  the  component  organizations  provides  for  a  more  efficient 
investment  management  process.  Tiered  accountability  has  been  embraced  across  DoD. 
This  includes  improving  DoD’s  ability  at  an  enterprise  level  to  maintain  the  appropriate 
level  of  visibility  into  the  component’s  operations. 

GAO  continues  to  be  a  valuable  and  constructive  partner  in  the  Department’s 
business  transformation  efforts.  The  recommendations  and  feedback  provided  will  help 
to  further  guide  DoD’s  process  of  continual  improvement.  We  welcome  GAO’s  insights 
and  look  forward  to  your  participation  in  our  future  efforts. 


Paul  A.  Brinkley 

Deputy  Under  Secretary  of  Defense 
(Business  Transformation) 
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GAO  DRAFT  REPORT  DATED  MARCH  30,  2007 
GAO-07-538  (GAO  CODE  310636) 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  institute  the  investment  boards,  including  assigning 
the  investment  boards  responsibility,  authority,  and  accountability  for  programs 
throughout  the  investment  life  cycle  and  specifying  how  the  business  investment 
management  system  is  coordinated  with  Joint  Capabilities  Integration  and  Development 
System  (JCIDS),  Planning,  Programming,  Budgeting,  and  Execution  (PPBE)  and  Defense 
Acquisition  System  (DAS)  to  ensure  that  well-defined  and  disciplined  business  system 
investment  management  policies  and  procedures  are  developed  and  issued,  (p.  34/GAO 
Draft  Report) 


POD  RESPONSE:  Partially-Concur  -  The  Department  believes  that  the 
IRB/DBSMC  process  and  tiered  accountability  with  the  Components  currently 
supports  accountability  for  programs  throughout  the  investment  lifecycle.  Further, 
the  Department  believes  a  linkage  currently  exists  between  the  IRB  certification 
and  review  processes  and  many  other  DOD  decision  support  processes  including 
JCIDS,  PPBE,  and  Acquisition,  as  depicted  in  the  figure  below  from  the  13 
December  2006  Business  Transformation  Guidance. 


This  linkage  is  also  addressed  in  the  IRB  Concept  of  Operations  (CONOPS) 
(previously  provided  to  GAO),  dated  29  August  2006,  in  section  7.2,  page  9.  To 
further  the  alignment  between  the  three  processes,  the  DoD  has  begun  to 
implement  the  Business  Capability  Lifecycle  (BCL)  concept  which  is  scheduled  to 
be  fully  implemented  by  FY08  and  included  in  the  DoD  5000  and  JCS  3170 
rewrites  scheduled  for  the  fall  of  FY08.  The  BCL  will  integrate  the  JCIDS  and 
DAS,  for  Tier  1  and  Enterprise  systems,  into  a  single  oversight  process  leveraging 
the  existing  IRBs  and  DBSMC  oversight  framework.  As  stated  in  the  March  2007 
Annual  Report  to  the  Congressional  Defense  Committees,  the  BCL  has  three 
phases: 
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•  Definition  -  The  BCL  approach  requires  the  PSA  and  the  functional 
sponsor  to  collaborate  to  identify  and  clearly  describe  the  root  cause  of 
a  business  problem,  long  before  a  vendor  is  involved  in  the  process. 
The  PSA  and  functional  sponsor  are  asked  to  clearly  explain  why 
solving  the  problem  will  benefit  the  Department  and  (importantly) 
validate  there  is  no  existing  solution.  This  problem  statement  and 
supporting  justification  become  the  basis  of  the  business  case  for  the 
proposed  capability,  which  will  be  reviewed  and  approved  by  the 
appropriate  IRB.  It  is  during  this  phase  of  the  BCL  that  the  Defense 
Acquisition  Executive  decides  whether  a  new  program  start  will  be 
approved  for  funding,  based  on  the  recommendations  of  the  IRB  and 
members  of  the  DBSMC. 

.  Investment  -  After  the  decision  is  made  to  fund  a  program  start,  the 
business  case  for  the  capability  is  expanded  by  the  functional  sponsor 
and  the  candidate  program  office  to  identify  the  scope  of  the  materiel 
capabilities  needed  to  solve  the  problem.  The  business  case  will  also 
define  the  desired  outcomes  for  the  capability,  including  objectives  and 
metrics,  solution  constraints  and  dependencies.  A  detailed  analysis  of 
alternatives  is  conducted  during  this  phase  and  included  in  the  business 
case  document,  which  is  augmented  by  a  proposed  acquisition 
approach  and  contracting  strategy. 

.  Execution  -  During  the  execution  phase,  responsibility  for  developing 
and  fielding  the  capability  is  formally  assumed  by  the  program 
manager.  However,  the  BCL  concept  requires  that  the  functional 
sponsor  remain  heavily  engaged  with  the  program  office  to  address  any 
issues,  requests  or  changes  to  the  scope.  In  particular,  the  BCL  requires 
that  the  functional  sponsor  re-validate  the  business  case  (including 
problem  definition,  expected  outcomes,  metrics,  and  costs)  before  each 
acquisition  milestone  or  investment  decision  point,  such  as  an  initial 
test  or  the  completion  of  the  definition  of  a  program  baseline. 

We  are  developing  specific  policy  guidance  to  amend  the  non-statutory  portions 
of  the  DoD  5000  senes  of  acquisition  regulations  and  the  JCS  3170  to  incorporate 
BCL. 

Under  Tiered  Accountability  and  as  system  owners,  Components  are  responsible 
for: 

•  Overseeing  program  progress  through  the  JCBDS  and  DAS 

•  Advocating  for  program  resources  in  the  PPBE  process. 

•  Coordinating  with  die  IRBs  when  system  certification  for 
development/modernization  is  required  at  key  milestones  in  the  Acquisition 
process. 

•  Managing  systems  that  are  past  the  developmenfimodemization  stage  through 
the  PPBE  process  and  the  annual  review  process  as  documented  in  the  IRB 
Guidance. 

The  IRB  CONOPS  and  the  IRB  User  Guidance  state  that  Components  are 
required  to  annually  review  all  business  systems,  including  those  that  are  in 
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sustainment,  su] 
process  such  as 
IRBs  review,  at 


gesting  that  they  perform  this  review  as  part  of  an  existing 
Re  annual  Program  /  Budget  Formulation  phase  of  PPBE.The 


million  dollars  as  required  by  the  FY2005  NDAA.  The  result  is  that  all  business 
systems,  whether  they  are  under  development/modemization  or  have  been  placed 
in  sustainment,  are  reviewed  annually  throughout  their  lifecycles. 


RECOMMENDATION  2;  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  select  new  investments,  including  specifying  how 
cost,  schedule,  and  benefit  data  are  to  be  used  in  making  certification  decisions;  defining 
the  criteria  used  to  select  investments  as  enterprise-wide;  and  establishing  consistent  and 
effective  guidance  for  business  enterprise  architecture  (BEA)  to  ensure  that  well-defined 
and  disciplined  business  system  investment  management  policies  and  procedures  are 
developed  and  issued,  (p.  35/GAO  Draft  Report) 

POD  RESPONSE:  Partially  Concur 

Partially  Concur 

The  BTA  has  defined  initial  criteria  for  selecting  enterprise-wide  investments  and 
is  in  the  process  of  applying  this  criterion  to  the  enterprise  systems  under  the 
Defense  Business  Systems  Acquisition  Executive  (DBSAE).  This  effort  is 
defining  a  framework  that  articulates  the  set  of  specific  characteristics  that  are 
appropriate  for  an  enterprise-level  solution. 

This  initiative  which  is  referred  to  as  “Rationalizing  the  Enterprise”  is  scheduled 
to  be  finalized  this  summer  and  will  be  incorporated  into  the  investment 
management  process  to  help  the  IRBs  and  Components  determine  which  business 
capabilities  should  be  implemented  at  the  Business  Mission  Area  (BMA) 
enterprise  level  versus  those  that  should  be  implemented  at  the  Component  level. 


Non-concur 

IRB/DBSMC  Policies  do  require  cost,  schedule  and  benefit  data  for  certification 
decisions  and  annual  review  IRB  assessments.  This  information  is  included  on 
both  the  annual  review  and  certification  dashboards.  Cost,  schedule  and 
performance  is  assessed  as  “green”,  “yellow”  or  “red”  based  on  specified 
thresholds  defined  in  policy  and  benefit  is  assessed  through  non-financial  and 
financial  metrics  substantiated  with  an  economic  viability  analysis.  IRB  decisions 
are  not  based  on  any  one  item  but  a  combination  of  factors,  some  of  which  are 
measurable,  and  some  less  tangible.  Cost,  schedule,  and  performance  are  the 
basis  upon  which  annual  reviews  are  conducted. 

Non-Concur 

BEA  Compliance  policies  were  released  April  10,  2006,  which  describe  the 
process  for  assessing  compliance  to  the  architecture  and  define  the  requirements 
for  an  architecture  compliance  plan.  This  guidance  has  also  been  enabled  through 
the  Architecture  Compliance  and  Requirements  Traceability  Tool  which  creates 
a  semi-automated  process  for  assessing  compliance  and  generating  a  Compliance 
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Plan.  It  also  provides  metrics  which  show  the  degree  of  alignment  to  the  BEA 
and  number  of  “compliant”,  “non-compliant”  and  “compliance  pending” 
instances. 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  reselect  ongoing  investments,  including  specifying 
how  cost,  schedule,  and  performance  data  are  to  be  used  in  the  annual  review  process  and 
providing  for  the  reselection  of  investments  that  are  in  operations  and  maintenance  to 
ensure  that  well-defined  and  disciplined  business  system  investment  management  policies 
and  procedures  are  developed  and  issued,  (p.  35/GAO  Draft  Report) 

POD  RESPONSE:  Non-Concur 

As  stated  above,  cost,  schedule  and  performance  data  are  used  in  the  annual 
review  process. 

Per  the  IRB  CONOPS,  dated  29  August  2006,  in  section  8.0,  page  13: 

•  Components  are  required  to  annually  review  all  business  systems, 
regardless  of  investment  Tier,  including  systems  for  which  there  is  no 
planned  development  or  modernization  spending. 

•  At  a  minimum,  as  part  of  the  annual  reviews  Components  should  make 
sure  that  systems  are  assessed  against  the  DoD  BEA,  ensure  systems  are 
included  in  the  Component  or  Enterprise  Transition  Plan,  and  that  all 
required  information  regarding  each  system  has  been  updated  in  the 
Department’s  global  business  systems  inventory. 

•  Components  are  required  to  submit  a  letter  to  the  IRBs  on  a  semi-annual 
basis,  on  a  schedule  consistent  with  the  Enterprise  Transition  Plan  update 
cycle,  listing  all  business  systems  that  have  been  reviewed.  These  internal 
Component  reviews,  coupled  with  notification  of  these  reviews  to  the  CA  / 
IRB,  meet  the  FY  2005  NDAA  annual  review  requirement. 


RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  integrate  funding  with  the  process  of  selecting  an 
investment,  including  specifying  how  the  Defense  Business  Systems  Management 
Committee  (DBSMC)  and  the  Investment  Review  Board  (IRB)  use  funding  information 
in  carrying  out  decisions  on  system  certification  and  approvals  to  ensure  that  well-defined 
and  disciplined  business  system  investment  management  policies  and  procedures  are 
developed  and  issued,  (p.  35/GAO  Draft  Report) 

DOD  RESPONSE:  Non-Concur 
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Funding  information  is  integrated  into  the  current  IRB/DBSMC  process;  funding 
information  for  every  investment  is  presented  to  the  IRB  membership  and 
documented  on  both  the  certification  and  annual  review  dashboards  and  PCA 
letters.  Funding  is  an  important  element  of  the  process  and  is  taken  into 
consideration  along  with  other  information  (e.g.  risk,  benefit)  during 
IRB/DBSMC  deliberations.  When  there  are  funding  issues  associated  with  a 
particular  investment,  they  are  addressed  during  the  IRB  process,  particularly 
during  the  annual  review  process.  If  they  are  related  to  poor 
management/execution,  the  IRB/DBSMC  may  recommend  reprogramming 
actions  to  support  better  alignment  of  budget  to  the  needs  of  the  portfolio.  Each 
IRB  decision  is  based  on  a  review  of  available  information  and  unfunded  requests 
are  handled  on  a  case  by  case  basis. 

RECOMMENDATION  5:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  oversee  information  technology  (IT)  projects  and 
systems,  including  providing  sufficient  oversight  and  visibility  into  component-level 
investment  management  activities  to  ensure  that  well-defined  and  disciplined  business 
system  investment  management  policies  and  procedures  are  developed  and  issued. 

(p.  35/GAO  Draft  Report) 

POD  RESPONSE:  Non-Concur  -  The  Department’s  investment  management 
process  for  business  systems  is  predicated  on  the  tiered  accountability  approach, 
under  which  DoD  Components  are  responsible  for  managing  their  IT  investments 
and  IT  portfolios  with  the  proviso  that  the  cognizant  IRBs  and  the  DBSMC 
provide  oversight  over  those  investments  to  ensure  compliance  with  10  U.S.C. 
2222,  as  added  by  Section  332  of  the  Ronald  W.  Reagan  National  Defense 
Authorization  Act  for  Fiscal  Year  2005,  and  other  applicable  laws,  regulations, 
and  policies.  Under  this  statute  the  IRBs  and  the  DBSMC  have  visibility  of  all 
systems  that  receive  in  excess  of  one  million  dollars  in  modernization  funding. 

The  Department  believes  the  GAO’s  recommendation  contradicts  the  tiered 
accountability  approach  in  recommending  that  the  Department,  from  a  corporate 
perspective,  oversee  Component  development  and  issuance  of  business  system 
investment  management  policies  and  procedures.  While  the  Department  does 
oversee  Component  business  system  investment  management  decisions  to  the 
degree  defined  in  the  IRB  CONOPS  and  has  issued  guidance  on  portfolio 
management  processes  to  the  Components,  in  accordance  with  tiered 
accountability,  it  does  not  guide  or  direct  the  Components  in  the  formulation  of 
the  Component-level  policies  and  procedures  by  which  their  investment  decisions 
are  reached. 


RECOMMENDATION  6:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  create  and  modify  IT  portfolio  selection  criteria  for 
business  system  investments,  (p.  35/GAO  Draft  Report) 
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POD  RESPONSE:  Partially  Concur  —  The  Department  continues  to  move  in 
the  direction  of  maturing  its  portfolio  management  processes.  Under  Tiered 
Accountability,  each  Component  is  responsible  for  developing  and  managing  its 
own  portfolio  management  process;  however,  when  it  is  in  the  best  interest  of 
DoD  for  a  portfolio  to  span  Components,  the  appropriate  IRB  can  establish  an 
“Enterprise  Portfolio.”  To  date,  DoD  has  stood  up  the  Distribution  Process 
Owner  (DPO)  Portfolio  which  looks  at  distribution  processes  and  supporting 
business  systems  across  all  DoD  Components.  The  DPO  is  chaired  by 
USTRANSCOM. 

With  the  implementation  of  BCL,  all  the  IRB  charters,  CONOPs,  and  Guidance 
are  under  revision.  The  revised  versions  will  clearly  articulate  the  criteria 
necessary  for  establishing  a  “Enterprise  Portfolio.” 

Additionally,  the  Department  has  implemented  the  Department  of  Defense 
Instruction  (DoDD)  81 15.01  -  “Information  Technology  Portfolio  Management”, 
which  defines  the  responsibilities  for  the  management  of  DoD  IT  investments  as 
portfolios  within  the  DoD  Enterprise  (to  include  Mission  Areas,  Sub-portfolios, 
and  Components). 

RECOMMENDATION  7:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  analyze,  select,  and  maintain  business  system 
investment  portfolios,  (p.  35/GAO  Draft  Report) 

DOD  RESPONSE:  Partially  Concur  -  The  Department  continues  to  move  in 
the  direction  of  maturing  its  portfolio  management  processes.  Under  Tiered 
Accountability,  each  Component  is  responsible  for  developing  and  managing  its 
own  portfolio  management  process;  however,  when  it  is  in  the  best  interest  of 
DoD  for  a  portfolio  to  span  Components,  the  appropriate  IRB  can  establish  an 
“Enterprise  Portfolio,”  To  date,  DoD  has  stood  up  the  Distribution  Process 
Owner  (DPO)  Portfolio  which  looks  at  distribution  processes  and  supporting 
business  systems  across  all  DoD  Components.  The  DPO  is  chaired  by 
USTRANSCOM. 

The  implementation  of  the  BCL  will  allow  the  IRBs  significantly  improved 
visibility  of  all  investments  being  made  in  given  portfolios.  Since  each  investment 
will  be  accompanied  by  a  business  case,  the  IRBs  will  have  the  opportunity  to 
make  investment  decisions  with  a  much  broader  set  of  criteria  than  is  possible  at 
the  current  time. 


RECOMMENDATION  8:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  review,  evaluate,  and  improve  the  performance  of  its 
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portfolio(s)  by  using  project  indicators  such  as  cost,  schedule,  and  risk.  (p.  35/GAO  Draft 
Report) 

POD  RESPONSE:  Partially  Concur  -  Under  existing  IRB  and  DBSMC 
process  and  procedure  IRBs  and  DBSMC  currently  review  cost  and  schedule  data 
as  part  of  the  investment  certification  and  annual  review  processes.  In  an  effort  to 
understand  project  risk  and  the  impact  of  risk  on  the  delivery  of  business 
capability  the  Department  has  implemented  the  Enterprise  Risk  Assessment 
Methodology  (ERAM).  ERAM  is  currently  being  executed  on  five  of  the  ten 
business  MAIS  programs.  The  output  of  the  risk  assessments  will  provide  an 
analysis  of  the  risks,  impacts  and  mitigation  strategies  for  given  portfolio 
investments  enabling  the  IRB  to  weigh  risk  impact  along  with  cost,  schedule  and 
performance  further  improving  investment  decisions. 

As  stated  in  the  March  2007  Annual  Report  to  the  Congressional  Defense 
Committees,  ERAM  is  a  collaborative  review  process,  bringing  the  functional 
sponsors,  the  program  office,  and  experts  from  the  acquisition  community 
together.  An  ERAM  team  begins  by  reviewing  existing  program  documentation, 
and  then  conducts  face-to-face  interviews  with  a  cross-section  of  key  program 
stakeholders  and  managers.  Based  on  this  information,  the  ERAM  team  evaluates 
program  risk  in  seven  key  areas  and  delivers  a  risk  mitigation  plan  as  quickly  as 
possible  (ideally,  within  five  to  six  weeks).  The  seven  risk  areas  are: 

•  Strategy 

•  Scope/Requirement 

•  Contract 

•  Technical 

•  People 

•  Process 

•  External 

The  quick  turnaround  is  important,  because  the  goal  is  to  give  the  sponsor  and 
program  manager  targeted,  actionable  advice  in  time  for  them  to  act  to  keep  the 
program  focused  on  delivering  capability. 

ERAM  adheres  to  DoD  Directive  5000  Series  principles  that  govern  Defense 
acquisition  activities.  Ultimately,  it  is  expected  that  ERAM  will  help  the 
Department  improve  its  acquisition  of  capabilities  by  achieving  several  key 
outcomes: 

•  Providing  the  right  information  needed  to  make  sound  optimized 
investment  decisions. 

•  Creating  a  clear  path  for  the  rapid  delivery  of  capability. 

•  Reducing  (or  removing)  burdensome  Overarching  Integrated 
Process  Team  (OIPT)  documentation  and  meeting  requirements. 
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•  Identifying  program  risks  early  enough  so  they  can  be  avoided  or 
mitigated. 

•  The  overall  vision  for  ERAM  is  to  provide  a  common  vehicle  for 
collaboratively  managing  program  risk  with  a  focus  on  rapid 
delivery  of  capability  at  reduced  cost  and  schedule. 

RECOMMENDATION  9:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  to  conduct  post  implementation  reviews  for  all 
investment  tiers  and  direct  the  investment  boards  who  are  accountable  for  corporate 
business  system  investments,  to  consider  the  information  gathered  and  to  develop  lessons 
learned  from  these  reviews,  (p.  36/GAO  Draft  Report) 

POD  RESPONSE:  Non-Concur  —The  Department  disagrees  that  this  process 
should  be  managed  by  the  Deputy  Secretary  of  Defense.  Requiring  the  Deputy 
Secretary  of  Defense  to  perform  post-implementation  reviews  is  redundant  with 
The  Office  of  Management  and  Budget  (OMB)  Circular  A- 130,  Chapter  8 
b.(l).(d)  that  requires  the  agency  “Conduct  post-implementation  reviews  of 
information  systems  to  validate  estimated  benefits  and  document  effective 
management  practices  for  broader  use.”  The  Department  will  capture  and 
leverage  the  lessons  learned  and  best  management  practices  from  these 
component  level  reviews  and  make  them  available  to  the  IRBs  and  across  the 
Components.  This  also  aligns  with  DoD’s  tiered  accountability  approach. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “Subscribe  to  Updates.” 
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A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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